[00:01] <smoser> harlowja, well, i fixed the ones i fixed
[00:01] <harlowja> lol
[00:02] <smoser> the 2 i fixed with ip
[00:02] <harlowja> thx :)
[00:02] <smoser> they are i'm pretty sure either
[00:02] <smoser> a.) you have no output of 'ip'
[00:02] <smoser> or (i think)
[00:02] <smoser> b.) you have not 'ip' in your path
[00:02] <smoser> because redhat typically does noth ave /sbin in path
[00:02] <smoser> but either way
[00:02] <smoser> the next one i think is just that have dns hijacking going on on that host
[00:02] <harlowja> could be
[00:03] <smoser> see is_resolvable
[00:03] <smoser> it tries to work around that stuff
[00:03] <smoser> but apparently not enough there.
[00:03] <smoser> so...
[00:03] <smoser> help me get a cent6 / cent7 container
[00:04] <harlowja> i only know VMs
[00:04] <harlowja> lol
[00:04] <smoser> well, i have a container
[00:04] <harlowja> VM
[00:04] <smoser> you just get it so things work
[00:04] <harlowja> :)
[00:04] <harlowja> agreed
[00:04] <harlowja> will get that in a few, templating cloud.cfg
[00:05] <smoser> well https://public.etherpad-mozilla.org/p/cloud-init-centos-unittest
[00:05] <smoser> right now a bunch of stuff fails for me
[00:05] <harlowja> kk
[00:05] <harlowja> paage still loading
[00:06] <harlowja> lol
[00:06] <smoser> i have a node up <IP_ADDRESS>
[00:06] <smoser> that you can go to ubuntu@
[00:06] <harlowja> k
[00:06] <smoser> and lxc is there...
[00:06] <harlowja> cubswin?
[00:06] <harlowja> lol
[00:06] <smoser> ssh keys.
[00:06] <harlowja> k
[00:06] <smoser> 101
[00:06] <smoser> is cubs win total
[00:06] <smoser> kind of like lol
[00:06] <harlowja> haha
[00:06] <harlowja> close enough
[00:08] <smoser> do you use python-mock from the distro ?
[00:08] <smoser> i think its too old.
[00:09] <harlowja> i usually run tox which then installs all the things
[00:09] <harlowja> so nothing from distro
[00:13] <smoser> hm.
[00:13] <smoser> well, i couldnt run tox
[00:13] <smoser> it didn't work either.
[00:13] <harlowja> hmmm, did u install tox ?
[00:13] <harlowja> how much did work?
[00:13] <smoser> from rpm ?
[00:13] <smoser> er.. yum
[00:13] <harlowja> or from pip
[00:15] <smoser> http://paste.ubuntu.com/23253475/
[00:15] <harlowja> ah, yes, pip install setuptools --upgrade also
[00:16] <harlowja> stupid old busted
[00:16] <harlowja> likely also pip install virtualenv --upgrade to
[00:16] <harlowja> i believe virtualenv has the bundled version of setuptools that tox then uses
[00:16] <harlowja> but i forget
[00:17] <harlowja>  pip install setuptools tox virtualenv --upgrade
[00:17] <smoser> still issues
[00:17] <smoser> checking that.
[00:17] <harlowja> k
[00:17] <harlowja> and keep on saying to yourself 'old stuff is more stable'
[00:17] <harlowja> lol
[00:17] <smoser> so essentially if this works, i'm basically running in tox
[00:17] <harlowja> ya
[00:18] <smoser> which...
[00:18] <smoser> protects me from most things
[00:18] <harlowja> *for testing
[00:18] <smoser> i'd like to run nosetests with the versions of things that are in the distro
[00:19] <harlowja> probably need to cap mock version then
[00:21] <harlowja> which is prob 1.0.1 on epel
[00:21] <harlowja> *epel 7
[00:21] <harlowja> and 0.8 on epel 6 :-/
[00:21] <harlowja> don't forget keep on saying to yourself 'old stuff is more stable'
[00:21] <harlowja> lol
[00:22] <smoser> well, i guess i dont care too much about the mock version
[00:22] <harlowja> ya, as long as it works, which it should
[00:23] <smoser> right. i'm fine to have newer of that, but if we're using a library we want to ouse the version in the distro.
[00:23] <smoser> i like that:
[00:23] <smoser>  yum install valid-package invalid-package valid-package2
[00:23] <harlowja> yum is sorta retarded...
[00:23] <smoser> goes and installs everything and then somewhere in the log it says 'invalid-package not available'
[00:23] <smoser> rather than jsut saying "um... can't do that"
[00:23] <harlowja> ya, i think the exit code is also 0 for that case
[00:23] <harlowja> from what i remember
[00:23] <harlowja> lol
[00:24] <harlowja> so u can't detect it in bash scripts
[00:24] <harlowja> (at least not via exit codes)
[00:26] <harlowja> hopefully dnf fixes that
[00:27] <smoser> FAILED (SKIP=28, errors=30, failures=1)
[00:27] <harlowja> nice, more than with tox it seems, lol
[00:27] <smoser> so thats in my cent6 after pip install and then tox
[00:27] <smoser> some recently regressed. :-(
[00:27] <smoser> TypeError: decode() takes no keyword arguments
[00:28] <harlowja> ah, ya, that one
[00:28] <harlowja> didn't i fix that
[00:28] <harlowja> i forget
[00:28] <harlowja> lol
[00:28] <smoser> how do you fix that ?
[00:28] <harlowja> cent6 and py26 are awesome
[00:28] <harlowja> i think u can just not give a keyword argument :-P
[00:28] <harlowja> and just do it positional argument only
[00:29] <harlowja> ya, just positional should be fine
[00:29] <harlowja> S.decode([encoding[,errors]]) -> object
[00:29] <harlowja> prob could do encoding=encoding in py27 only or something
[00:29] <smoser> oh. right.
[00:29] <harlowja> 'old stuff is more stable'
[00:29] <smoser> ok. with taht change, now
[00:29] <harlowja> lol
[00:29] <smoser> FAILED (SKIP=28, errors=3)
[00:30] <harlowja> cool
[00:30]  * powersj thinks we should probably get CI unit tests going again as they fail right now
[00:30] <harlowja> whats the last 3
[00:30] <harlowja> 'old stuff is more stable' 'old stuff is more stable' 'old stuff is more stable'
[00:30] <harlowja> lol
[00:31] <smoser> http://paste.ubuntu.com/23253529/
[00:31] <smoser> k. first is easy enough (content)
[00:33] <harlowja> ya, the rest are dict comphrenhsions
[00:33] <smoser> set comprehension on one of them :)
[00:33] <harlowja> oh ya
[00:33] <harlowja> ya, just turn those into set(iterable) and dict(iterable)
[00:33] <harlowja> and that's all those become
[00:34] <harlowja> so not so bad
[00:34] <smoser> http://paste.ubuntu.com/23253540/
[00:35] <harlowja> cool u can even do
[00:35] <harlowja> set(m['uri'] for m in f['apt']['primary'])
[00:35] <harlowja> if u really care
[00:35] <harlowja> either will be fine
[00:35] <harlowja> no need for intermediary list
[00:36] <smoser> oh.
[00:36] <smoser> how is that
[00:36] <smoser> oh. ididnt know you coudl
[00:37] <smoser> i like that
[00:40] <harlowja> :-p
[00:40] <harlowja> m['uri'] for m in f['apt']['primary'] is a generator
[00:41] <smoser> yeah, but only if wrapped in parens
[00:41] <smoser> $ python -c 'm for m in (1,2,3)'
[00:41] <smoser>   File "<string>", line 1
[00:41] <smoser>     m for m in (1,2,3)
[00:41] <smoser>         ^
[00:41] <smoser> SyntaxError: invalid syntax
[00:41] <smoser> $ python -c '(m for m in (1,2,3))'
[00:41] <smoser> happy
[00:41] <harlowja> right
[00:41] <harlowja> so u put it in parans
[00:41] <harlowja> ha
[00:41] <harlowja> set( )
[00:41] <harlowja> lol
[00:42] <smoser> it is kinda wierd like that
[00:42] <smoser> that the parens to the function call suffice
[00:42] <smoser> that kind of hurts my brain
[00:43] <harlowja> ya, its probably a weirdness in the python syntax that allows for it
[00:45] <smoser> how should i say "what version of centos am i on"
[00:45] <smoser> ie, i want to know 6 or 6
[00:45] <smoser> or 7
[00:47] <smoser> alright. good.
[00:48] <harlowja>  in /etc/redhat-release i think
[00:48] <harlowja> or lsb_release i think has something
[00:49] <harlowja> or python has some stuff u can use
[00:49] <smoser> https://public.etherpad-mozilla.org/p/cloud-init-centos-unittest
[00:50] <smoser> so... given that patch above, i can run tox in cent6 or cent7
[00:50] <smoser> thats good, but ideally we'd be able to run 'nosetests' against distro-installed versions (as would be found in a runtime)
[00:50] <harlowja> ya, that may require a little more version tweaking
[00:51] <smoser> and the 'build' thing mostly worked... at least used to.
[00:51] <smoser> so, tahts good. thanks harlowja
[00:51] <harlowja> ya, there is a file or 2 that is missing for brpm
[00:51] <harlowja> but i'm hoping with nrezinorn that brpm goes away
[00:51] <smoser> you seem my comments https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/305882
[00:51] <smoser> i have to run
[00:51] <harlowja> ya
[00:51] <smoser> thanks for your help jxharlow
[00:51] <harlowja> whos that
[00:51] <harlowja> lol
[00:52] <smoser> you changed your middle name when you moved to godaddy
[00:52] <harlowja> JXMenHarlow
[00:52] <smoser> is that because harlowja.coolguy was taken?
[00:52] <smoser> but you could still get harlowjx
[00:52] <harlowja> nah, i asked and they just said, meh that's what u got
[00:52] <harlowja> i asked about harlowja, and it seemed like alot of work
[00:52] <harlowja> so i just gave up
[00:52] <harlowja> lol
[00:53] <harlowja> https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+ref/kill-brpm (where brpm goes away)
[00:53] <harlowja> it seems to work on cent7 at least, ha
[00:53] <harlowja> nrezinorn just wants a spec file, lol
[00:53] <smoser> but i like brpm
[00:53]  * harlowja runs away
[00:53] <harlowja> lol
[00:54] <smoser> as long as some way we can make an rpm
[00:54] <harlowja> ya
[00:54]  * smoser out
[00:54] <smoser> later
[15:39] <smoser> harlowja, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/307333
[15:39] <smoser> what do you think about that ?
[15:54] <avsh> Hi, I am using vmware with openstack
[15:54] <avsh> Virtaul CD-Rom which used for config drive is not removed after Machine is provisioned
[15:54] <avsh> Leaving admintrative password and other data as clear text in ( DVD-DRIVE config-2 )
[15:54] <avsh> what configuration is need for cloud init to remove the drive when vm is provisioned?
[15:55] <avsh> can anyone please help me with my query?
[16:03] <smoser> avsh, do you have non-root users who can mount a cdrom ?
[16:03] <avsh> yes
[16:04] <smoser> to my knowledge, cloud-init couldn't on its own rid the system of that drive. you could 'eject /dev/cdrom'
[16:04] <smoser> and then it would not be there, but possibly a 'eject -t /dev/cdrom' would pull back in the tray and have it again
[16:04] <smoser> and if it is Read-only media, then cloud-init can't write it to blank it
[16:05] <smoser> can you give the file that has the password in it ? it'll help me diagnose where to look to see what esle could be done.
[16:11] <avsh> ec2/
[16:11] <avsh> openstack/2012-08-10/
[16:11] <avsh> openstack/2013-04-04/
[16:11] <avsh> openstack/2013-10-17/
[16:11] <avsh> openstack/content
[16:11] <avsh> openstack/latest/meta_data.json
[16:11] <avsh> openstack/latest/user_data
[16:11] <avsh> openstack/latest/vendor_data.json
[16:11] <avsh> This is the Folder Structure
[16:11] <avsh> and openstack/latest/meta_data.json has contents like { "admin_pass": password, "random_seed": "*******" }
[16:20] <smoser> oh. well, good for you that cloud-init doesn't care what is in there
[16:20] <avsh> smoser, let me know if you need any info
[16:20] <smoser> it ignores it
[16:20] <smoser> as on linux, ssh keys are preferred.
[16:21] <avsh> but users can see other sensitive information which is a security concern
[16:21] <avsh> like if i install a software on the vm, that software password is an example
[16:25] <smoser> avsh, do you have a reason to let users mount that disk ?
[16:25] <smoser> why not just remove users from the 'cdrom' group
[16:26] <avsh> we can do that. we don't see this issue with Openstack + KVM
[16:26] <avsh> only with Vmware + Openstack
[16:27] <avsh> trying to understand, it is configuration with cloud-init or vmare nova driver
[16:27] <smoser> just because you're not looking in the right place :)
[16:27] <smoser> i suspect the same data is availble in http://<IP_ADDRESS>/openstack/
[16:28] <smoser> and any malicious user already knew that.
[16:28] <avsh> ok
[16:28] <smoser> cloud-init can route off the that particular address so that only root would be able to get at it
[16:29] <smoser> with
[16:29] <smoser>  disable_ec2_metadata: true
[16:30] <avsh> ok, let me check on kvm instance with the url you provided
[16:40] <avsh> smoser, you made my day
[16:40] <avsh> you are correct, I am not seeing at right place with kvm + openstack
[16:40] <avsh> It can access all the data with the above url
[16:40] <avsh> I can rule out vmware opensstack nova drive
[16:41] <smoser> avsh, alternatively you can do things in a different way.
[16:42] <smoser> you can use '#include-once' to include other cloud-config things... and make those expiring or one-time-read urls
[16:42] <smoser> the metadata services are not intended to be secure
[17:00] <avsh> smoser, I will check on the #include-once, thanks
[17:45] <smoser> rharper, around ?
[17:45] <smoser> i've 2 things for you... one. do you have a readthedocs.org account (or can you get one).
[17:45] <smoser> 2.  http://paste.ubuntu.com/23256482/
[19:03] <rharper> smoser: here
[19:03] <rharper> I have a rtd account
[19:03] <rharper> lemme get on (2)
[19:04] <rharper> what am I looking at with (2) ?
[19:06] <smoser> what is rtd account ?
[19:06] <smoser> i will share access to cloud-init project
[19:06] <rharper> ah, I see , a slow ish boot; total time is 11 seconds though
[19:06] <rharper> read-the-docs
[19:07] <rharper> ah, right
[19:07] <smoser> rharper, yeah....
[19:07] <smoser> um more interesting than that.
[19:07] <rharper> raharper
[19:07] <smoser> that was a 2+ minute boot
[19:07] <smoser> :)
[19:07] <rharper> 17:32:37 to 17:32:56
[19:07] <rharper> it's not cloud-init log
[19:08] <smoser> yeah.
[19:08] <rharper> that's like 20 seconds wallclock
[19:08] <smoser> thats what is fun
[19:08] <rharper> so something else (look at systemd-analyze blame
[19:08] <rharper> can I haz ssh to ami ?
[19:08] <smoser> i think clock is moving backwards
[19:08] <rharper> oh, ntp!
[19:08] <smoser> you can... yeah
[19:09] <smoser> let me set access up for you through my bastion
[19:09] <smoser> i assume its reproducible on serverstack
[19:09] <smoser> but
[19:09] <rharper> ok
[19:10] <rharper> dmi data /sys/class/dmi/id/product_name returned OpenStack Nova  -- that log is not from AMI on EC2 ..
[19:11] <smoser> right
[19:16] <rharper> if it's kernel related, it's possible that it could be reproduced in sstack; however, given that the virt layer is going to handle memory differently (booting xen on ec2, vs kvm on openstack) that may mean we won't reproduce the same amount of slowdown;  the kernel but that's reference has to do with SLAB/SLUB config and other changes
[19:22] <smoser> rharper, ssh-via<EMAIL_ADDRESS><EMAIL_ADDRESS>[19:22] <smoser> first ip is my bastion. i set you up to jump through there.
[19:22] <smoser> second is the system.
[19:22] <smoser> ssh-via is http://smoser.brickies.net/git/?p=tildabin.git;a=blob;f=ssh-via;
[19:23] <rharper> blob_plain is what I want
[19:27] <rharper> smoser: in
[19:29] <rharper> Sep 30 17:31:33 ubuntu systemd[1]: Time has been changed
[19:29] <rharper> something *did* reset the clock
[19:30] <rharper> Sep 30 17:31:33 ubuntu systemd-timesyncd[556]: Synchronized to time server <IP_ADDRESS>:123 (ntp.ubuntu.com).
[19:30] <rharper> Sep 30 17:30:42 - prior event
[19:31] <rharper> Sep 30 17:31:33 - time has changed
[19:31] <rharper> so, slow clock moved forward by just under a minute
[19:32] <rharper> and then, ntp syncs it and moves it another minute forward, Sep 30 17:32:23
[19:33] <rharper> brb
[19:37] <smoser> rharper, its wierd though that cloud-init's logging didn't see that.
[19:37] <smoser> rsyslog must be doing it ? and it keeping its own clock or something?
[19:56] <rharper> smoser: it's in journctl
[19:56] <rharper> the time change happened async from cloud-init execution
[19:56] <rharper> not quite sure how journctl keeps track of time vs. python logging/rsyslog
[19:58] <rharper> smoser: note the odd delta between the entry timestamp (Sep 30 17:32:37, vs the timestamp collected for the welcome message: at Fri, 30 Sep 2016 17:30:39 +0000)
[19:59] <smoser> yeah. its wierd.
[19:59] <smoser> and systemd is confused by this
[19:59] <smoser> as it *does* say cloud-init took 2 minutes to run
[19:59] <smoser> when i'm pretty sure watching a wall clock that is not the case.
[20:00] <rharper> correct
[20:00] <rharper> I was going to do the relative time between events in ci and I'm positive it didn't take all that time (rather we've got a clock jump)
[20:01] <rharper> I suspect if you uncloud-init data , reboot
[20:01] <rharper> it won't be as a long
[20:01] <rharper> I wonder why the VM clock is so far off though
[20:01] <rharper> 2 minute adjustment is pretty large
[20:06] <rharper> you launched me at: Fri, 30 Sep 2016 17:29:59 +0000
[20:06] <rharper> kernel booted     : Fri, 30 Sep 2016 17:30:30 +0000
[20:06] <harlowja> smoser interesting if u haven't seen it
[20:06] <harlowja> https://cloud.google.com/compute/docs/containers/vm-image/#using_cloud-init
[20:08] <smoser> rharper, yeah, i agree.
[20:09] <rharper> so, I don't know what to do unless timectl could tell us how much time was adjusted
[20:09] <rharper> it seems like it's a systemd bug too
[20:09] <smoser> really sucks
[20:09] <rharper> since analyze didn't update (acknowledge that ntp changed time)
[20:09] <rharper> if it knows the timedelta, it could apply that
[20:09] <smoser> in cloud-init we could read uptime
[20:09] <rharper> to provide true time
[20:10] <rharper> despite clock shift
[20:10] <smoser> theres a way to get that i think rather thatn /proc/uptime
[20:10] <smoser> (althoguh /proc/uptime is mocked in a container for us, and a kernel interface probably isnt)
[20:11] <rharper> right
[20:13] <smoser> https://github.com/xmonader/linuxsysinfo/blob/master/sysinfo.py
[20:15] <rharper> yeah, the procfs is slightly slow in container
[20:15] <rharper> the cloud-final message is always high on blame due to reading sysinfo
[20:16] <rharper> it's relatively slow to other modules
[20:19] <smoser> rharper, well, we could read via sysinfo.
[20:19] <rharper> syscalls, yeah; but that'd be host data, right ?
[20:19] <rharper> sorta like dmesg
[20:19] <smoser> probalby could even read once from /proc/uptime
[20:19] <rharper> it's just not right
[20:19] <smoser> and then count that offset
[20:19] <smoser> :)
[20:19] <rharper> right, proc/uptime once and caching that would be useful
[20:20] <smoser> and then form then on out ask the sysinfo
[20:20] <rharper> but the issue is the 4 invocations of cloud-init
[20:20] <smoser> well, 4 reads of /proc/uptime is probably "not that bad" in the grand scheme of all the bad things.
[20:20] <rharper> but from exec to exit, we could do sysinfo for time;  but honestly; I think rdtsc is likely faster for absolute cycles
[20:20] <rharper> smoser: it's the hottest thing left on lxd reboots
[20:20] <smoser> well then.
[20:21] <rharper> it's not bad at all
[20:21] <smoser> :)
[20:21] <rharper> we're at 0.25 second reboot
[20:21] <rharper> but it's still roughly 30% of that in cloud-final message
[20:21] <rharper> so, if we could reduce that , then we'd see sub .2 second reboots
[20:22] <rharper> rdtsc would be faster than call to proc due to fuse
[20:28] <rharper> hrm, the monotonic clock jumps too when time is set; I suppose that's expected... but that sounds wrong
[20:37] <rharper> smoser: so, can we see if /var/lib/systemd/clock exists in the cloud-image ?
[20:38] <rharper> ah, it doesn't (at least the lxd rootfs doesn;'t have it)
[20:39] <smoser> as in is it created at runtime you mean ?
[20:39] <smoser> versus already present?
[20:39] <rharper> right
[20:39] <rharper> timesyncd uses that as a restore clock to this as soon as it starts (if it exists)
[20:40] <smoser> oh.
[20:40] <rharper> that could be a jump but it would have been backwards quite a bit
[20:40] <smoser> restore from what ?
[20:40] <rharper> each sync with ntp, that file is updated with the last good stamp from ntp
[20:40] <rharper> https://lists.freedesktop.org/archives/systemd-devel/2015-May/031988.html
[20:40] <rharper> It
[20:40] <rharper>   implements sNTP and will sync the last known time to disk every time
[20:40] <rharper>   it gets an sNTP sync or the system is shut down.
[20:40] <rharper> At boot it uses
[20:40] <rharper>   that time to reinitialize the clock, as early as possible, before
[20:40] <rharper>   NTP is done. THis will give you monotonic time which should solve
[20:40] <rharper>   your probelm.
[20:41] <rharper> so, the journal, and other loggers on the system use CLOCK_MONOTONIC, which is susceptable to NTP changes in clock (always forward)
[20:41] <rharper> so, AFAICT, this is just a really out of sync clock on the host
[20:49] <smoser> which *is* susceptable ?
[20:49] <smoser> or is not succeptable
[20:50] <rharper> is
[20:50] <rharper> CLOCK_MONOTONIC_RAW is not
[20:51] <smoser> hm.
[20:51] <smoser> harlowja, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/307363
[20:52] <smoser> powersj, that gives us a way (via the gist linked there) to run uni tests in centos6 pretty easily
[20:58] <smoser> rharper, raharper is now a maintainer of CloudInit in readthedocs
[20:58] <smoser> and fyi, magicalChicken's doc fixes are there now!
[20:59] <smoser> (it had stopped building when we moved from bzr)
[21:00] <rharper> smoser: cool, i see it
[21:01]  * smoser has to run
[21:02] <smoser> harlowja, if you could look at that... you can merge it if you want
[21:02] <smoser> it seems nice.
[21:02] <harlowja> kk
[21:02] <harlowja> cools
[21:02] <smoser> later.
[21:03] <powersj> smoser, very cool
[21:03] <harlowja> woah
[21:03] <harlowja> nice nice
[21:03] <harlowja> all the modules got filled in???
[21:03] <harlowja> sweet
[21:04] <harlowja> that only took a couple years to finish, lol
[21:04] <harlowja> thx guys! :)
[21:10] <jgrimm> magicalChicken, ^^
[21:14] <jgrimm> harlowja, smoser: so in the gce link, it mentions that they've implemented setting UID. Any particular reason that hasn't been implemented in base cloud-init up to now besides priorities?
[21:14] <harlowja> i forget
[21:14] <harlowja> i thought i remember a patch for that
[21:14] <harlowja> lol
[21:14] <jgrimm> bug for it. https://bugs.launchpad.net/cloud-init/+bug/1396362
[21:15] <jgrimm> yeah, i remembered seeing it in the backlog
[21:15] <harlowja> unsure about that one
[21:15] <harlowja> why is it a diff file
[21:15] <harlowja> lol
[21:15] <harlowja> did they not sign the CLA
[21:15] <harlowja> weird
[21:15] <jgrimm> when i saw it in the gce doc, reminded me that i'd seen it requested before
[21:15] <harlowja> ya
[21:22] <powersj> smoser, I setup cloud-init to run unit tests 2x a day now across the architectures. It will git clone master, figure it is better than noting.
[21:23] <powersj> I can figure something out for the new centos one too when I get back home
[21:24] <powersj> and since it does not respond to merge requests, but just does master, it will email us on failures, hence the mail you probably just got