To enhance security on GitLab, we should require 2FA. Right now, 2FA is optional.
GitLab can use a TOTP for its 2FA, for which apps are widely available.
Let's communicate to the community that we'd like to do this effective Saturday, Oct 1 2022.
More info on TOTP, from Mikael's message below:
TOTP does not require a physical key. It can be, but it's not mandatory.
The second factor can be time-based, one-time passwords (TOTP). This require the storage of a seed in a app (often a mobile app), but some password managers like Bitwarden, 1password, or Dashlane also support it. There are also command line application that are able to generate those TOTP:
If you want to use a physical key, you just need to ensure that it's compatible with the WebAuthn (aka FIDO2) protocol. There are many available options on the market for compatible keys. I can personnaly recommend the SoloKeys (https://solokeys.com), a product built with open source hardware and software.
2FA does not require a physical key. It can be, but it's not mandatory.
The second factor can be time-based, one-time passwords (TOTP). This require the storage of a seed in a app (often a mobile app), but some password managers like Bitwarden, 1password, or Dashlane also support it. There are also command line application that are able to generate those TOTP:
If you want to use a physical key, you just need to ensure that it's compatible with the WebAuthn (aka FIDO2) protocol. There are many available options on the market for compatible keys. I can personnaly recommend the SoloKeys (https://solokeys.com), a product built with open source hardware and software.
I am having second thoughts on the "enforcing" part of this.
In his May 2022 blog post[1], Mike Milinkovich has stated: One thing that is clear, however, is that simply putting the burden of added security work on the shoulders of our committers and project leaders is not an option.
I feel that enforcing 2FA on GH and GL is a heavy-handed approach that goes against that statement, and given some of the recent backlash towards some of the proposed changes to the infra (migrating off Gerrit/Bugzilla, moving project websites), I am incredibly hesitant to use the terms "enforce" and a specific target date for such enforcement.
Agreed. I'm thinking about the following approaching:
Broadcast as much as we can (crossprojects, committers ML, discuss with marketing about other means) that we invite all committers to activate 2FA on their account and consider enforcing it for their projects. They would reach out to us for project-level enforcement.
For the projects who don't reach out voluntareely, we reach out to them individually to convince them that this is important. They still decide about the timeline but we inform them that we may, at some point, enforce it.
Once all of this is done, we can decide if we want to enforce it and when. If we ever reach this point, this should not be a surprise for anybody in the community and the pushback should be minimal.
+1 to this, but I think we also need to get committers to evangelize this with their contributor communities. I believe 2FA will not only affect committers, but as we can only reach out to committers, we need committers to talk to their contributors.
I'm beginning to understand the impact only now: It seems that even the occasional bug reporter needs 2FA for each and every action: reporting a bug, answering a question during bug triage etc.? If that's the case I wonder how many folks out there, who might be willing to help if efforts are low, will just leave when they find out about this.
To be fair, GitHub is already requiring 2FA for most (if not all) users. So, 2FA seems to become the de-facto standard for creating a user accounts almost anywhere.
I don't think there is a viable alternative (let 2FA be optional?) or middle ground (require 2FA only for code contributions).
@tiagolucas is expected to conduct some tests to re-validate this; however, the way we have currently designed the enforcement is that only contributors with projects on gitlab.eclipse.org must activate 2FA on gitlab.eclipse.org. There's no requirement for casual bug reporters to do so.
Aforementioned testing was conducted with an Eclipse account with no committer role on any project. It's tested that this account can create an issue on a repository under Eclipse Projects without being prompted for 2FA and without being required 2FA after doing so (I was afraid that it would get "Reporter" role or similar after creating the ticket).
It's tested that this account can create an issue on a repository under Eclipse Projects without being prompted for 2FA
That's good news, thanks!
I reality I had to perform six steps, just to be allowed to write this response:
provide username
provide password
find and unlock my phone (sim was already unlocked :) )
unlock the 2FA app
copy the 2FA token
review the gitlab email about this log in
Whenever the 2FA app insist in providing its password (rather then biometrics), it will require another password to unlock the place where that password is stored.
I don't want to sound too negative, but I wonder if the consequence of all this will be that people will no longer lock their phones, no longer use different (or any) passwords for all those apps and indirections to secrets, etc.
[W]e’re announcing that GitHub will require all users who contribute code on GitHub.com to enable one or more forms of two-factor authentication (2FA) by the end of 2023.
From my experience it depends a lot on the implementation of it all. Does it require that you pick up your phone, unlock it, go to an app, unlock the app, approve, etc, every time you close the browser and open it again? Then it quickly becomes irritating and delays things a lot if you have to do this dozens of times a day. However, if it allows trusted devices, where it will only be required to use 2FA to reconfirm once a month, or when you're suddenly on an IP address of a different country, or so. Then it is a different story entirely. I don't have any experience with 2FA of GitLab yet, so I'm not sure how obtrusive it is.
I also think people will have to just try it out (myself included), to get some experience, maybe get used to it (or not, if it is too obtrusive), etc.
However, if it allows trusted devices, where it will only be required to use 2FA to reconfirm once a month, or when you're suddenly on an IP address of a different country, or so.
That's pretty much how it works. I only get asked about 2FA every once in a while or if I switch devices, actively log out, etc.
That is not what I get. If I close my browser, open it again, and go to the EF GitLab again, I'm logged out again. I have to log in with my EF credentials, and provide a 2FA code. This is super annoying!
OK, but why force this 2FA on us, when the 2FA is too obtrusive. I asked about this before, and was promised it would not be too obtrusive. Now it is. I think the known issue should be fixed before forcing 2FA. Now it breaks the flow too often, it is too obtrusive.
I understand that the user experience is essential, and any disruptions can be challenging. However, the issue you've pointed out is independent of whether 2FA is activated or not. Activating 2FA neither exacerbates nor alleviates the issue with session persistence across browser restart in gitlab. Also, this is an issue with gitlab the software, not the way gitlab.eclipse.org is configured.
The primary purpose of enforcing 2FA for all committers is to significantly enhance our security. While we strive to optimize the user experience, security remains a top priority.
Activating 2FA neither exacerbates nor alleviates the issue with session persistence across browser restart in gitlab.
Before, I would indeed need to log in again with EF account credentials. But, my browser stores that. It is an annoying click on 'Sign in', but that's it (1 click to sign in, one to confirm). With 2FA, it is much more steps. It then requires picking up my phone, unlocking, going to the authenticator app, unlocking that app, typing over the code, and clicking the 'Verify code' button. I go from 2 to 9 steps. So it is exacerbated.
Also, this is an issue with gitlab the software, not the way gitlab.eclipse.org is configured.
https://gitlab.com/gitlab-org/gitlab/-/issues/25361 is closed, and seems to be more about redirecting back to the requested page after login. It does have some pointers to other pages/issues, that I've read (see below).
https://gitlab.com/gitlab-org/gitlab/-/issues/121569 indeed seems on topic. It is very long, but I read it anyway. It has some interesting information (see below). But the issue got addressed by adding a 'Remember me' checkbox to the sign in page, and has since been closed and locked. That checkbox is however not there for the EF GitLab (see also below).
At https://gitlab.com/gitlab-org/gitlab/-/issues/121569 there is the suggestion to let the browser reset the session when the browser is closed and started again (chrome://settings/onStartup). This works, but I don't like that behavior, personally. Also, on my work laptop, I can't set that setting, as some of the browser settings are controlled by the company.
I found https://docs.gitlab.com/ee/user/profile/#cookies-used-for-sign-in, which indicates certain cookies could be set to keep users logged in. Maybe the EF GitLab does not set these (correctly). However, I do see quite some active sessions at https://gitlab.eclipse.org/-/profile/active_sessions. I also see the _gitlab_session cookie. Maybe the EF login coupling does not integrate well with the GitLab cookies? Maybe because I have to log in with my EF account credentials each time, also each time a new session is created, and 2FA is also re-requested?
I found that the _gitlab_session, ECLIPSESESSION, and ECLIPSE_ENV cookies are set to expire with the session. known_sign_in is set to expire in two weeks. If while logged in, I set all these cookies to some date in the future, and close the browser, start the browser again, and visit the EF GitLab again, I'm still logged in. This points to the cookies not having the right expiration set.
Thank you @ddennis for this deep investigation. There is definitely a couple of things that should be tried to improve the user experience on gitlab.eclipse.org. May I suggest you move your comment to #65 and we continue the discussion over there? Thanks!
In the videos, "1 password" and the "Google Authenticator" mobile app are used, but it works very similar with other password managers or authenticator apps.
Hmm. With "ratio of 2FA enablement" you are referring to the members of the orgs, correct? Yes, you do.
I tried to pull the number of users with 2FA disabled from the GitHub organization API and got ~980 users, which would be a much lower enablement ratio.
Enable All users in this group must set up two-factor authentication
Disable Subgroups can set up their own two-factor authentication rules
Do the above settings 30 days (and as such set the "delay 2FA enforcement" to 24*30=720 hours) prior the actual enforcement so that committers have a grace period before being forced to activate it.
Target date: October 24th (the week after after Eclipse Con, a chance to help some committers during the office hours there)
I have tried to enable 2FA authentication on my account, but it says that my current password is invalid, which is blocking me from completing the set up. I'm using my LDAP password (the one I use to sign in, which I know works). Is there some other password I need to be using?
It seems that at some point your account was associated with a gitlab "local" identity (ie not connected an EF account) and it triggers this password request from gitlab when you want to setup 2FA. The requested password is not your Eclipse Foundation account password, hence why it does not work.
To fix it, please go to https://gitlab.eclipse.org/-/profile/password/edit and ask for a password reset (this will be only for the "local" identity, not your EF account). Once your "local password" is changed, you should be able to use it in the field where it's being requested.
This "local identity" password does not need to be remembered beyond this step as it's not used anywhere else.
Same for me. For the record, I did have to reset the password since I didn't have it anymore, but that worked out just fine.Same for me. For the record, I did have to reset the password since I didn't have it anymore, but that worked out just fine.
We've never had a "local identity", but @tiagolucas found other references to GitLab sites with this same issue; it appears it's not specific to Eclipse. We'll follow up with GitLab.
Creates a new user. Note only administrators can create new users. Either password, reset_password, or force_random_password must be specified. If reset_password and force_random_password are both false, then password is required.
force_random_password and reset_password take priority over password. In addition, reset_password and force_random_password can be used together.
We use the force_random_password flag since there is no way for us to retrieve a plain text password of an Eclipse account for obvious security reasons.
I've proposed that we investigate utilizing the reset_password flag for creating new accounts. I am hoping this could streamline the process for new committer accounts since we would trigger the password reset for them (The email they receive should have information about this. Let's hope that GL allows us to modify the text).
However, it's not clear to me if enabling LDAP will solve this problem.
We would need to test if the LDAP integration replaces the local Gitlab password. If not, then my suggestion for using the reset_password flag might be the best thing we could do...
Created a ticket in each (non-archived) repository under Eclipse Projects to warn project committers about the forthcoming enforcement of 2FA, with the text present here, the following being the most important:
If the form asks you for a password in order to set up 2FA on your account, this is not your Eclipse account’s password. It is a known bug on Gitlab that some accounts are requested a “local” password despite having one in the Active Directory.
You should request a password reset and use that same password for this form. This process does not change your Eclipse account password.
Do I need to purchase a hardware token for account access?
No. GitLab supports two 2FA methods:
Time-based One Time Password (TOTP) compatible with mobile apps like Google Authenticator or Authy, and several password managers such as Bitwarden or 1Password.
WebAuthN, which necessitates a hardware token, typically a USB key (examples include Solo 2 key or Yubikey). These tokens are sometimes referred to as FIDO2 keys.
In the near future, 2FA will become mandatory for authentication on your accounts. Should you not have enrolled by the deadline we communicated to you, access to the platform will be restricted.
I already have 2FA enabled on gitlab.eclipse.org, do I need to do anything?
No, you’re all good.
What do I do if I lose my 2FA device?
We highly recommend the utilization of diverse secondary authentication methods. In the event that you misplace all your secondary authentication elements, recovery codes will be the only way to restore account access. By securely storing your recovery codes, you'll ensure the ability to regain access.
Note that the Eclipse IT team may be able to recover access to accounts with 2FA enabled if both the 2FA credentials and account recovery methods are lost. This will require extra identity verification and direct contact with security@eclipse-foundation.org or webmaster@eclipse-foundation.org.
Looked into some of this a little to get us some options. There is an ability to update some of the email templates, as they are just files on the server. By default, they are available under /opt/gitlab/embedded/service/gitlab-rails/app/views/notify/ with text and HTML templates for the supported email messages. The most promising message would be the new user message, where we could add some context to mention the need to reset password to setup 2FA.
If we go this route, we'd have to track the email templates somewhere and add the application of them to the server in our playbook for updates, as they will be wiped away on system updates.
Is it possible to remove the random passwords for accounts automatically generated by the sync script? The objective is to end up with accounts identical in terms of configuration compared to accounts that are created when (non committers) users first logs into GitLab with an Eclipse account. I understand this might require direct modifications to the GitLab database.
If this is possible, then I'd not change anything to committers' account creation, but "only" incorporating the additional step to remove the randomly assigned password.
Creates a new user. Note only administrators can create new users. Either password, reset_password, or force_random_password must be specified. If reset_password and force_random_password are both false, then password is required.
force_random_password and reset_password take priority over password. In addition, reset_password and force_random_password can be used together.
We need to pick 1 of the 3. We don't have access to the user password so password is definitely not an option. We've been using force_random_password so far but it's not working for us.
I am currently recommending that we try reset_password which I believe will prompt the user to set a password on login.
I understand this might require direct modifications to the GitLab database.
I don't have access to the database but I suspect we should be able to answer that question if we compare a user that was created with the API vs one that was created via openid_connect.
I have reviewed the recommendations, but there is still some ambiguity concerning the two distinct issues that need attention:
The procedure for establishing new accounts for incoming committers.
The rectification of already existing accounts for current committers.
Regarding the first point, I advise against employing the reset_password option to avoid user confusion. Instead, we should persist with the force_random_password option. However, the automatic committers account creation procedure needs to be updated to mirror the end results of a committer accessing gitlab.eclipse.org with an Eclipse Account for the first time. The solution is, probably, to include a step that modifies the database in a manner that essentially nullifies the password.
As for the second matter, which should be our top priority, we must resolve the issues affecting all committer accounts established so far. As we do not have yet the exact solution, it requires investigation by either the infra or webdev team. Discovering and implementing a solution will not only rectify current issues but also help with ironing out the kinks for fixing the automatic committer account creation from the first point.
Do we want to grant the sync script write access to the GL database or do we prefer to create something separate to avoid that?
If offering a good user experience to committers necessitates it, I view it as an inevitable step.
Regarding the two options you've outlined, considering that by the end of the year all committers will be required to enable 2FA on GitLab, I don't believe asynchronous tasks (option (2)) provide a viable solution. This approach would result in a delay between the creation of the account on GitLab and when a user can set up 2FA without being prompted for a password. Such a delay could lead to user confusion. Therefore, I would recommend opting for option (1).
From a security standpoint, the sync script should have the minimal privileges necessary within the database. This includes limiting access to the machine through firewall rules (IP allowlisting) and using credentials that have no permissions other than updating the specific column we aim to modify.
I have been trying to get 2FA working for 4 weeks now, and it still doesn't.
The mentioned "detailed instructions" just explain one happy flow with a phone device. That flow just plain fails for me with "invalid pin" for about 15 times now, so I don't buy it's the pin code.
Switching to a yubikey, I read some stuff (much more than I wanted) to get access to it at my system. Then I assumed the "detailed instructions" cover that setup too, but they don't (scanning a QR code with a yubikey?).
So I have to figure out everything from scratch as I have never done this, and have no interest in security as topic.
Thinking I could use a FIDO2 key with webauth, but then I read you first need 2FA before you can add that and ... yes, joy.
I think my last option is to use the yubikey for TOTP, which apparently is possible.
While it may solve my access, it still kills my current way of working which involves usage from 2 different computer systems. More pain expected.
Getting very close to the point of just giving up. I can't focus on other things anymore, and loosing sleep on it.
It's not worth it.
Greetings @ahofkamp, I've been helping others through this workflow. If the PIN is failing, there could be a difference of more that 15 seconds in the time configured in the device. This also takes into account your configured timezone. Using the TOTP with Yubikey may not solve this problem if you're using the same device (Yubikeys that I know of do not possess a clock).
Regarding the use of a TOTP when you can't scan the QR code, you'll have a section titled "Can't scan the code?" just below the QR itself with a "Key". I fully sympathize with not wanting 2 computer systems to access anything, so I always use the key to setup TOTP.
I ditched the phone as feasible option 2 weeks ago already, so I already steered clear of the "use same device" problem luckily.
Last week I tried the key, but didn't have all information that I needed so I aborted the attempt. Yesterday I tried again, and it failed. After a few tries I thought of deleting the account entry at the key and start from scratch. That did the trick.
So while i works now, the whole experience is just horrible. Tools are lying to you, and they (probably) leave partial validated entries in an invalid state. There is no guidance beyond one happy flow with a phone. After that, you just have to figure out what things are, how all the various parts are related to each other in the jungle of TLA security abbreviations, and what form that takes for your own system.
Fwiw, one way to setup a key for Linux (Ubuntu derivative) is:
Install pcscd (smart card deamon, it finds the key and makes it available to applications).
To make pcscd work, either start it manually sudo pcscd -f, or set it as a service in systemd, with sudo systemd enable pcscd.service
The application yubioath-desktop is an authenticator (sudo apt install yubioath-desktop).
For actual 2FA activation:
Start pcscd, and yubioath-desktop.
Insert the key.
Authenticator should see the key.
Click an add account button.
It asks for (iirc) "issuer", "name", and a code. The former two are simple strings used for display afaik (but I know very little of these things). The "code" entry is the text of the fallback when scanning is not possible, displayed near the QR code in the GL verification page.
Then fill the remaining fields at the verification page (current password, and the 6 digits that the authenticator is showing.
Keep fingers crossed for extra luck when you press the "verify" button, and hopefully it works.
There is also an authenticator application by Yubico that looks usable, didn't try it.
Last but imho not least, don't trust error messages to tell the truth so you can reason how to proceed.
For those who had issues similar to @mojavelinux (#1251 (comment 1224595)), and were waiting for it to be fixed before activating 2FA, I'm glad to say that a patch has been deployed. All committers to date should be able to activate 2FA without having to apply the workaround described in #1251 (comment 1224689).
This morning I wanted to add an answer to some discussions in the project. It asked for a 2FA key !!
It's silly, since when is typing a few lines of text in a public project a major security risk???
Something is very very wrong here.
As a committer to a project, your authentication grants you write access to the code repository, necessitating enhanced security through 2FA. Regrettably, I'm unaware of any systems, including GitLab, that enable you to specify your intended action during authentication to tailor security policies accordingly, whether it's for commenting on a ticket or editing a file. Moreover, even if such a feature were available, I doubt it would offer a user experience that is particularly pleasant.
Sometimes I don't push any code for at least 1/2 a week. Even if I do, it's always in feature branches, they are always just proposals for a change that are reviewed and discussed.
Much less than once a week I press the "merge" button. Even then it's in develop, and not in a built product that ends up with users.
The frequency and processes employed by your project are insufficient to counter the risks associated with potential compromise of your account, particularly considering the elevated access privileges granted to a committer.
However, it's noteworthy that the mandatory code reviews do mitigate some risks in the threat model of an open-source project (see https://slsa.dev/spec/v1.0/threats-overview and https://slsa.dev/spec/v1.0/threats for details — be sure to check on future directions as well for complementary threats). Please continue this practice as it's an excellent measure. Unfortunately, this serves only as a secondary line of defense.
I can appreciate you want to check it's me when pushing super-powers buttons. However at the low frequency of doing that, the "always 2FA" requirement means that in 80-90% of the days I do all the 2FA juggling for nothing, as I am not going to press any super-button that day.
Compare that with how sudo works at a Linux system. I am a normal user at the system, so I can't by default do much harm to the system itself (I can only make a mess in my own data). If I want to do something potentially dangerous to the system, I must first type sudo in front of the super-powers command. Then and only then it asks for a check it's me, and if ok, it performs the command.
I agree the problem is much more complicated, we're just too much in love with automagicking things :D
I have initiated the enforcement of 2FA for all projects hosted on gitlab.eclipse.org/eclipse. Throughout this week, there will be several periods of activation and deactivation (referred to as 'brownouts') of this enforcement. Full enforcement will commence from next Monday.