[Jenkins-infra] Pagerduty on-call modifications

[Olblak]

Changes are applied, so we shouldn't be notified during the night
anymore
---
-> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
---




On Tue, Aug 7, 2018, at 10:15 PM, Kohsuke Kawaguchi wrote:
> Thanks, this is helpful. What you just described is a runbook of its
> own, I suggest you capture that in jenkins-infra/runboooks README.md
> or something.> 
> I'll follow this when the next pager come in. Now I'm kinda looking
> forward to seeing one... :-)> 
> On Tue, Aug 7, 2018 at 12:04 PM Olblak <me at olblak.com> wrote:
>> __
>> We try to  explain fixes for recurring issues on that (private)
>> repository -> https://github.com/jenkins-infra/runbooks>> And if we don't know what to do, we raise a Jira ticket and assign to
>> the person who has the information.>> A recent example was the last spamming session on #jenkins-infra and
>> #jenkins-community irc channel (INFRA-1732).>> We raised a ticket and Tyler filled the documentation and now the
>> next they come, we'll be ready  (ง •̀_•́)ง>> It's important to mention when something is not clear otherwise we'll
>> have to deal with the bus factor :)>> And if for any reasons, we realise the PD notification is important
>> then we still have contact information on that repository as well.>> 
>> ---
>> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
>> ---
>> 
>> 
>> 
>> 
>> On Tue, Aug 7, 2018, at 6:00 PM, Arnaud Héritier wrote:
>>> yes same than KK
>>> I'm happy to help but with the move to Azure I'm completely lost in
>>> how I could help (I'm even not sure to still have the required
>>> rights/permissions)>>> If I have the instructions of what to do (or not do :-P) I'll
>>> continue to help with pleasure>>> 
>>> On Tue, Aug 7, 2018 at 4:57 PM Kohsuke Kawaguchi <kk at kohsuke.org>
>>> wrote:>>>> Thanks for taking the lead on this. I think your plan makes sense.>>>> 
>>>> I'd like to use this opportunity to ask how I should respond to
>>>> these PD notifications. Most of them I get is indeed disk space
>>>> issues, but I have no idea how to resolve that, so mostly I just
>>>> ignore them.>>>> 
>>>> On Tue, Aug 7, 2018 at 6:13 AM Olblak <me at olblak.com> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> I would like to update the Pagerduty rotation, currently every
>>>>> people on-call are assigned to the same group and rotated every 6
>>>>> hours which leads people after a while to be on call in the middle
>>>>> of the night while others are available.>>>>> We don't get a lot of notification (disk space issue most of the
>>>>> time), but I find it useless to be on-call in the middle of the
>>>>> night and not being notified when I am in front of a computer
>>>>> during the day.>>>>> 
>>>>> Now I suggest a "follow the sun" strategy with four groups and
>>>>> depending on your timezone (or preference) you are assign to a
>>>>> specific groups.>>>>> 
>>>>> It would be something like
>>>>> 
>>>>> Group 1 (5AM to 11 AM): Arnaud, Daniel, Olblak (7AM UTC+2 to 1PM
>>>>> UTC+2)>>>>> Group 2 (11AM to 5PM): Arnaud, Daniel, Olblak (1PM UTC+2 to 7PM
>>>>> UTC+2)>>>>> Group 3 (5PM to 11PM): Andrew,Tyler, Kohsuke (10AM UTC-7 to 4PM
>>>>> UTC-7)>>>>> Group 4 (11PM to 5 AM): Andrew, Tyler, Kohsuke (4PM UTC-7 to 10PM
>>>>> UTC-7)>>>>> 
>>>>> The global strategy would remain the same
>>>>> 
>>>>> Level1 notification sent to on call people
>>>>> After 30minutes
>>>>> Level2 notification are sent to Tyler/Daniel/Olivier
>>>>> After 30minutes
>>>>> Level3 notification is sent to  KK
>>>>> 
>>>>> Because we have at least three people per group, you will only be
>>>>> on-call during one slot per day?>>>>> Shouldn't we review on call people annually to be sure that they
>>>>> are still available?>>>>> 
>>>>> Any suggestions?
>>>>> 
>>>>> ---
>>>>> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
>>>>> ---
>>>>> _______________________________________________
>>>>> Jenkins-infra mailing list
>>>>> Jenkins-infra at lists.jenkins-ci.org
>>>>> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>>>> -- 
>>>> Kohsuke Kawaguchi
>>>> _______________________________________________
>>>> Jenkins-infra mailing list
>>>> Jenkins-infra at lists.jenkins-ci.org
>>>> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>>> 
>>> 
>>> -- 
>>> -----
>>> Arnaud Héritier
>>> http://aheritier.net
>>> Mail/GTalk: aheritier AT gmail DOT com
>>> Twitter/Skype : aheritier
>> 
> -- 
> Kohsuke Kawaguchi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20180808/69834ce4/attachment.html>