Those of you using Jenkins and Autopkg
//www.gstatic.com/images/branding/product/1x/groups_32dp.png
Why does it need to return anything other than what it already would do for a single recipe? You chain a number of recipies together. If there is an update to do you on any of them fire the whole thing off again with the run flag (is that not what your Jenkins does). If it fails, you check why it failed? Don't forget the verbose output would still be there, so either if running it directly from the command prompt you still see the exact same output to why it failed. And, furthermore, Jenkins captures this in it's log. Any email report would contain this and you'd easily be able to see what and why it failed. You'd then be able to have Jenkins respond to the exit code in a number of ways depending on the code?

If it failed on recipe 3 of 5, what difference does the completion of recipe 1 or 2 have on the entire job? Your emailed the console log (or see the red dot in jenkins). Reading that you find that jobs 1 & 2 are ok, 3 failed on the url. You fix the url and on the next scheduled run everything goes through. It exits with 0 that all went well?

I see where your coming from. I just think that the plist method makes it overly complicated. Maybe autopkg should only accept single recipes at a time for exit code to be given? I suppose I look at it in an old fashioned light where when you put something into a process you should get something meaningful back from the process. ie. at the moment you get nothing really useful other than an exit code of 1 or 0. This is useless from any scripting process unless your reading the verbose output. At least with defined exit codes for a single recipe you can deal with result of any number of failures via simple scripting. ie. run the update or send a failure email to systems or send an email failure to build team.

Tell me if I'm wrong about Tim's script... But from how I read it it doesn't give responses based on individual packages? It checks *all* packages from the recipes you give it. Then downloads any updates, then triggers a build of *all* of them if it finds anything? So any output from here is also pretty useless as you'd need to read the returning log by hand to check each went in?

As in Case 4 above, what happens with his script? Does this script (the entire thing) run for just one recipe at a time? How can it pass anything else useful about a list of recipes other than it ran and triggered a build? Are we not back to the same point?

Maybe we need a different script to anaylise the plist that Tim's script generates via autopkg? So a check script runs after Tim's script (via jenkins) which calls more jenkins tasks based on the packages that need updating via autopkg?

Et voila? A much longer set of scripts to essentially do what autopkg could do with a single recipe in 1 line?

Ok, so tell me if I'm reading this all wrong. Please do. I'm new to jenkins, autopkg and only a 5 month year old with munki. But I have a single task that updates my autopkg recipes. When this completes it fires off an autopkg task for each recipe that I'm interested in. One task for Skype, one task for VLC etc etc. When all of these tasks has completed I have another task which is fired that rebuilds the catalogue.

Is this about right so far? Currently I have to look at jenkins every day to see if something has gone wrong. Which it did when Firefox got updated 2 days ago. What I'd like is for me to be able to have jenkins email me when a package gets updated or email me when it fails. For both I would like to know what package we're talking about. So running these tasks individually makes a lot of sense. Checking the exit code of the check script tells me a few things. If there is an update, or if something erroneous went wrong and where it went wrong. If *Every* package send me a failure email, I'm going to be looking pretty hard at our internet connection or the hard drive space! If just one fails I can look straight at it.

So, maybe I am missing something?

I'm so glad to be finished for the weekend, I need the rest!

A.

So far I'm of the opinion that, at least by default, AutoPkg should reserve nonzero exit codes for when AutoPkg itself has uncaught exceptions or errors. For example, an error exit code is raised when you do a 'repo-add' and it can't locate a Git binary, or you mis-use a command option. This is similar to how other utilities on UNIX and OS X behave.

By default, Jenkins actually will not proceed with build steps if they exit non-zero, which you're probably aware of by now. I'm sure there are ways around that, but they most likely involve either installing a new plugin (which might necessitate another plugin) or some other kind of hack, as this deviates from the idea of what Jenkins is typically used for. There are some things in Jenkins that can be surprisingly challenging to work around if doing non-standard "build software and test" tasks without getting your hands pretty dirty.

When Per wrote AutoPkg, he built in pretty robust error handling from the beginning, which is what allows AutoPkg to run and collect error info from failing recipes into its final output report (either what's printed to screen or with --report-plist, or in the output receipt plist written to disk).

I also think there _could_ be the argument to have an _option_ to exit with error codes instead, but what event "category" (new download, new import, recipe error, etc.) trumps another when multiple recipes are being run? Everyone who would use such an option would make different assumptions, need to consult yet more documentation that tries to explain the hierarchy of error code precedence, etc. I think the argument for optional nonzero exit codes makes much more sense when you can only run a single recipe.

You mentioned earlier: "You'd then be able to have Jenkins respond to the exit code in a number of ways depending on the code?"

Do you know how you would do this (I don't)? You can write more shell scripts to try to make Jenkins work exactly how you'd like it to, but if you want to do this in the context of using the ScriptTrigger plugin you'll still likely need to defer that logic further down the build step with some other kind of wrapper that parses something.. and your best bet then is the machine-parsable output of --report-plist or the receipt file. And, you're back full circle, with more scripts, parsing the output and deciding what to do based on it. The idea of "more wrapper scripts to do what AutoPkg could do in one line" might apply for this specific scenario of yours and how you'd imagine exit codes to work, but it will definitely cause confusion and isn't even easy to explain, as this discussion thread indicates.

This is all part of why there isn't one single "this is how you use AutoPkg with Jenkins" guide, even though some people have been great to document how they've set it up in their environment. There's also definitely more information that could be added to the report plist - for example, it might make sense (though would be a breaking change) to structure the output so that it can be keyed by, or at least include, the recipe identifier, making it easy to pick out which recipes have new downloads and run only these.

The script I've posted, as the documentation states, is a rough demonstration of how you could parse the report plist to do the dumb thing of continuing on with the build if there's a new download.

How you would then decide what recipes to run, and how to run them, is up to you. I would probably start by simply running the same recipe list without --check and not worry about the fact that it's re-running other recipes. This is why the Munki import processor is idempotent - you can re-run the recipes and not keep re-importing the items to the repo.

Jenkins is one tool for doing centralized automated tasks, but it's not the only one, and it is a CI environment after all, so it's heavily biased towards those types of workflows. BuildBot is another popular server app which, I gather, is much more flexible because it works more like a Python library and isn't configured via a webapp. So you might come closer to your ideal using that, but it certainly will require you to write more code. You can also simply write a LaunchDaemon and pipe the output to sendmail given when certain keywords are present in the console output.

In the end, I don't think you're going to get around needing to craft something that glues some pieces together to fit your needs. I think that's what we all do, because people's needs and expectations vary.

-Tim

Hi Tim,

Thank you for such an in depth and considered reply.

I've only started working with Jenkins since Greg's presentation (as is probably apparent from all the above!), so please don't feel offended by the comments I have made. They are in no way intended to be critical of the work you have put into these developments. I see you mention "some people have been great to document how they've set it up in their environment". This is actually the part I have struggled most with and probably where my difficulties begin. I feel I have been setting this up in line with what others have been doing. I've search this group and the munki-dev group and not really found much in the way of other people set ups.

We have a really fast internet connection here so I shouldn't be too worried about the bandwidth used, but I suppose this is essentially always on my mind. For me, running a full check followed by another full download seemed a bit wasteful. That said, it probably doesn't use that much bandwidth. I'm going to spend some more time over Christmas considering our best of scenario. Maybe we don't need to know if something gets updated?

Any how, if your States bound, happy Thanksgiving. If not, like me, have a good weekend.

Anthony.

I did something similar using email-ext: email-ext is setup to send an e-mail on both success and failure status (determined by Jenkins) along with the log file and then if it's a successful build BUT it does nothing (no download or import) it cancels the send by setting cancel = true in the pre-send script (not the template). Simple groovy script grep of the log file (had to be simple - I don't know java, groovy or python - I'm a hack)

For the life of me I couldn't get setting cancel = true to work in the email template, but the same exact code works beautifully in the pre-send.

So you get a copy of the log file on a failure, and the log file on a successful import or download, and no email if the job build ran successfully but did nothing.

I'll see about posting this tonight on the group and then getting something setup on GitHub. It's pretty simple code but it wasn't easy to figure out how to stop spamming myself with successful "nothing done" emails due to being unable to find documentation on the cancel = true not working in the template.

I'm a n00b with Jenkins/AutoPkg but the trip has been wonderful so far since others have blazed a pretty easy to follow trail : we now have an auto-updating Mac software repository share that also happens to single-source with our munki_repo/pkgs (that notifies when new stuff is download and imported or failed, that with Jenkins AD-auth and https is secure and has a clean and easy to use interface making it easy to see when something's gone wrong and diagnose :-)

M@

Sent from my iThneed