[Jenkins-infra] Pagerduty on-call modifications

[Kohsuke Kawaguchi]

Thanks, this is helpful. What you just described is a runbook of its own, I
suggest you capture that in jenkins-infra/runboooks README.md or something.

I'll follow this when the next pager come in. Now I'm kinda looking forward
to seeing one... :-)

On Tue, Aug 7, 2018 at 12:04 PM Olblak <me at olblak.com> wrote:

> We try to explain fixes for recurring issues on that (private) repository
> -> https://github.com/jenkins-infra/runbooks
> And if we don't know what to do, we raise a Jira ticket and assign to the
> person who has the information.
> A recent example was the last spamming session on #jenkins-infra and
> #jenkins-community irc channel (INFRA-1732).
> We raised a ticket and Tyler filled the documentation and now the next
> they come, we'll be ready  (ง •̀_•́)ง
> It's important to mention when something is not clear otherwise we'll have
> to deal with the bus factor :)
> And if for any reasons, we realise the PD notification is important then
> we still have contact information on that repository as well.
>
> ---
> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
> ---
>
>
>
>
> On Tue, Aug 7, 2018, at 6:00 PM, Arnaud Héritier wrote:
>
> yes same than KK
> I'm happy to help but with the move to Azure I'm completely lost in how I
> could help (I'm even not sure to still have the required rights/permissions)
> If I have the instructions of what to do (or not do :-P) I'll continue to
> help with pleasure
>
> On Tue, Aug 7, 2018 at 4:57 PM Kohsuke Kawaguchi <kk at kohsuke.org> wrote:
>
> Thanks for taking the lead on this. I think your plan makes sense.
>
> I'd like to use this opportunity to ask how I should respond to these PD
> notifications. Most of them I get is indeed disk space issues, but I have
> no idea how to resolve that, so mostly I just ignore them.
>
> On Tue, Aug 7, 2018 at 6:13 AM Olblak <me at olblak.com> wrote:
>
>
> Hello,
>
> I would like to update the Pagerduty rotation, currently every people
> on-call are assigned to the same group and rotated every 6 hours which
> leads people after a while to be on call in the middle of the night while
> others are available.
> We don't get a lot of notification (disk space issue most of the time),
> but I find it useless to be on-call in the middle of the night and not
> being notified when I am in front of a computer during the day.
>
> Now I suggest a "follow the sun" strategy with four groups and depending
> on your timezone (or preference) you are assign to a specific groups.
>
> It would be something like
>
> Group 1 (5AM to 11 AM): Arnaud, Daniel, Olblak (7AM UTC+2 to 1PM UTC+2)
> Group 2 (11AM to 5PM): Arnaud, Daniel, Olblak (1PM UTC+2 to 7PM UTC+2)
> Group 3 (5PM to 11PM): Andrew,Tyler, Kohsuke (10AM UTC-7 to 4PM UTC-7)
> Group 4 (11PM to 5 AM): Andrew, Tyler, Kohsuke (4PM UTC-7 to 10PM UTC-7)
>
> The global strategy would remain the same
>
> Level1 notification sent to on call people
> After 30minutes
> Level2 notification are sent to Tyler/Daniel/Olivier
> After 30minutes
> Level3 notification is sent to  KK
>
> Because we have at least three people per group, you will only be on-call
> during one slot per day?
> Shouldn't we review on call people annually to be sure that they are still
> available?
>
> Any suggestions?
>
> ---
> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
> ---
> _______________________________________________
> Jenkins-infra mailing list
> Jenkins-infra at lists.jenkins-ci.org
> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>
> --
> Kohsuke Kawaguchi
> _______________________________________________
> Jenkins-infra mailing list
> Jenkins-infra at lists.jenkins-ci.org
> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>
>
>
> --
> -----
> Arnaud Héritier
> http://aheritier.net
> Mail/GTalk: aheritier AT gmail DOT com
> Twitter/Skype : aheritier
>
>
> --
Kohsuke Kawaguchi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20180807/77b2e9df/attachment-0001.html>