SoMee May 28 Live Blog of Deployment

So, it's May 28th, and we are doing the big deployment migration! wooo, scary, fun, sad, frustrating, but hey that's our job!

Keep an eye here, for our changes, and when we do them, so you all can be kept up to date with where somee.social and beta.somee.social are in the progress.

NOTE: Our devs will update here when they can, but we'll try to keep it as active as possibly can!

Our Plan

If you want a highlevel of what we are doing:

1. DONE : Update beta.somee.social with latest may 28 release
   1a: DONE: Test beta.somee.social
2. DONE: Take beta.somee.social and somee.social offline
3. DONE: Take a snapshot of beta database and beta post objects and move them to a backup
4. DONE: Migrate production data from somee.social (and database) to the beta service
5. DONE: Re-enable beta.somee.social
6. DONE: Test make sure it works
7. DONE: Move somee.social to archive.somee.social
8. DONE: Move beta.somee.social to somee.social
9. DONE: Re-enable somee.social
   IN PROGRESS: 9a: Test somee.social
10. PENDING: Re-enable archive.somee.social
    10a. Test archive.somee.social
11. Celebrate
12. Get some wine
13. PENDING: Sleep

Live Updates

• 5/28/2021 11:37 pm: We have begun the process of deploying our staging server to beta.somee.social to test our May 28th release (ensure it works properly there), we have the new web socket system deployed, and are working on the backend and frontend deploys as well. As a side note: Until we take beta.somee.social down, if you have an account there, you'll get a sneak peek of what's in the update!
• 5/28/2021 11:57 PM: We have created the short video buckets, permissions, and configurations, and are working on the frontend configurations to make short videos workable as well once we do the frontend deploy.
• 5/29/2021 12:26 AM: We have successfully created the ingress controllers for beta, created CloudFront distros for the new short videos, and updated all the caching and header information. Now working on the backend releases before we update beta.somee.social
• 5/29/2021 12:51 AM: We have given our beta node a bit more resources - and are simply testing the infrastructure for the backend deployment (gotta make sure it doesn't run out of juice w/ all these video on demand things, ya know :P)
• 5/29/2021 1:05 AM: We have deployed the new backend, working on the frontend and micro-services
• 5/29/2021 1:38 AM: We have deployed beta.somee.social and sted short vids, found some issue, but nothing that will directly stop us just yet.
• 5/29/2021 1:45-2:20 AM: We have tested beta.somee.social, and all main tests have passed - we do see one error with ShortVideos on Firefox not working - because of the encoding. Our team will do a hotfix within a day to fix that issue.
• 5/29/2021 2:32 AM: We have taken somee.social offline, and will be taking beta.somee.social offline
• 5/29/2021 2:47 AM: We started the database and data migration process for backup purposes, to ensure nothing is lost on beta during the transition. This can take time, as there are literally tens of thousands of objects to backup.
• 5/29/2021 4:24 AM: We are still working on the above task! As we said, it can take some time to migrate all the data. In the meantime, we are also working on moving the deployment of somee.social over to archive.somee.social. This process takes about an hour as well for each environment (as we have to update many configuration files). Will keep updating here as we make progress - but it may be an hour or two for the next update 🙂 (note also, Phillip lost internet as Las Vegas is silly about doing work on the internet this late at night, but he is up on mobile web and we still have our devops team plugging away even w/o him 😛 🙂)
• 5/29/2021 6:15 AM: We are finishing up on the configuration changes to move SoMee and Beta - it's a long process team 🙂 However, we have finished the data backups and are still doing the database restore. No worries, we are making slow, but steady progress 🙂
• 5/29/2021 8:15 AM: We ran into a technical issue that prevented us from moving beta.somee.social over, but we are working in the Infrastructure team to address this.
• 5/29/2021 9:00 AM: The infrastructure team has met with us and we have agreed to a plan moving forward and are working on implementing it.
• 5/29/2021 9:35 AM: We have implemented the infrastructure team's plan, and the frontend for somee.social is now online, we are just working with the DevOps team to work on the backend.
• 5/29/2021 10:25 AM: We are still working on migrating all the DevOps side for the backend, but the India team ran into a severe thunderstorm and lost power. The entire building is out of power, and they are not able to work right now.
• 5/29/2021 10:30 AM: We have started the AMA, and are still waiting on the India team to get back power.
• 5/29/2021 12:18 pm: We have finished the AMA (a very long one today), but the India team is still without power. We will have an extended downtime due to this.
• 5/29/2021 12:20 PM: Met with stakeholders and marketing and advised of delay; weather cannot be avoided, and our CTO has been up since 2 am yesterday coordinating all of this. He needs sleep, and the India team will need to have slept as well as it's 12:49 am there. They will continue in the morning, and Phillip will continue in about 8 hours.
• 5/29/2021 5:51 PM: DevOps team got the backend online for somee.social, and are running the database migrations to ensure it's working with the latest features. The site may appear online, and you can login, but some things will NOT be available until those migrations are run.

UPDATED ETA: 6:30pm PST (worst case scenario)

Note from Phillip/CTO:

Again we apologize for the extended outage, but rest assured, it will be online this evening if not sooner. Our goal here is to make sure that everything is stable and ready, and this type of migration, while we thought we could do it in 5-6 hours, turned into a technical challenge due to how the initial development team (you know who I mean) coded things, and hooked it up to the infrastructure - those accounts couldn't be used to migrate, and we couldn't maintain our plan to move with their infrastfducture needs in place. So we have adjusted that, worked with Onica to get the main frontend services back online, and will work with the DevOps team once they have sleep and power, to get the remaining backend services online.

So, we do apologize for the ongoing outage, but it's almost online for somee.social. We are just running a few more database migrations - right now Followers/Following are not returning for some users, so we are looking into that. However, the newsfeed should be good - as should groups, companies, and photos. Posting, and commenting should be fine as well.

Issues Affecting Release:

• Backend isn't migrated yet (we must migrate each of the main backend services and pods to coorelate with the new domain names for somee.social and archive.somee.social)
• Phillip has been up since 2 am yesterday - almost 48 hours
• India team has no power, and they have been working this for nearly 12 hours as well
• Followers/Following is Offline due to a database migration issue - we are fixing the script
• Archive.somee.social is still offline as we are rebuilding the indexes for it - the old site will be online soon.

Plan for Addressing Issues:

• India team will return to work bright and early, and kick off the changes for the needed backend services (about an horus work)
• India team will ensure the data buckets are migrated over (they were kicked off, and sync was running, just have to make sure it transferred those milliosn of objects over)
• India team will ensure that the backend service is running for somee.social
• India team will replace the placeholder index.html (maintenance doc) with the proper index.html so the stie works again
• India Team will confirm that Symfony and Backend are online for Followers/Following
• The development team will test the site to ensure no issues

Regards,

--Phillip Rumple/CTO