TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Real-world stories of how we’ve compromised CI/CD pipelines

265 点作者 usrme超过 3 年前

14 条评论

staticassertion超过 3 年前
Thank you for writing this up.<p>Some thoughts:<p>1. Hardcoded credentials are a plague. You should consider tagging all of your secrets so that they&#x27;re easier to scan for. Github automatically scans for secrets, which is great.<p>2. Jenkins is particularly bad for security. I&#x27;ve seen it owned a million and one times.<p>3. Containers are overused as a security boundary and footguns like `--privileged` completely eliminate any boundary.<p>4. Environment variables are a dangerous place to store secrets - they&#x27;re global to the process and therefor easy to leak. I&#x27;ve thought about this a lot lately, especially after log4j. I think one pattern that may help is clearing the variables after you&#x27;ve loaded them into memory.<p>Another I&#x27;ve considered is encrypting the variables. A lot of the time what you have is something like this:<p>Secret Store -&gt; Control Plane Agent -&gt; Container -&gt; Process<p>Where secrets flow from left to right. The control plane agent and container have full access to the credentials and they&#x27;re &quot;plaintext&quot; in the Process&#x27;s environment.<p>In theory you should be able to pin the secrets to that process with a key. During your CD phase you would embed a private key into the process&#x27;s binary (or a file on the container) and then tell your Secret Manager to use the associated public key to transmit the secrets. The process could decrypt those secrets with its private key but they&#x27;re E2E encrypted across any hops between the Secret Store and Process and they can&#x27;t be leaked without explicitly decrypting them first.
评论 #29969962 未加载
评论 #29971647 未加载
评论 #29972528 未加载
评论 #29971595 未加载
MauranKilom超过 3 年前
Interesting to learn that credentials in environment variables are frowned upon. I mean, makes sense if your threat model includes people pushing malicious code to CI, but aren&#x27;t you more or less done for at that point anyway? If &quot;legitimate&quot; code can do a certain thing, then malicious code can do too. I guess you&#x27;ll want to limit the blast radius, but drawing these boundaries seems like a nightmare for everyone...
评论 #29965849 未加载
评论 #29965885 未加载
评论 #29972650 未加载
评论 #29965844 未加载
lox超过 3 年前
We’ve been using Sysbox (<a href="https:&#x2F;&#x2F;github.com&#x2F;nestybox&#x2F;sysbox" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;nestybox&#x2F;sysbox</a>) for our Buildkite based CI&#x2F;CD setup, allows docker-in-docker without privileged containers. Paired with careful IAM&#x2F;STS design we’ve ended up with isolated job containers with their own IAM roles limited to least-privilege.
评论 #29967138 未加载
评论 #29965948 未加载
INTPenis超过 3 年前
Many of these points are about running pipelines in privileged containers. Something I actually took extra time to resolve for my team. That&#x27;s when I discovered kaniko first, and shortly after podman&#x2F;buildah.<p>After that podman and buildah have gotten a lot of great reviews from people so I think they&#x27;re awesome.<p>For an old time Unix sysadmin it just doesn&#x27;t make sense to run something as root unless you absolutely have to.<p>Which also makes the client excuse in the article so strange, they had to run the container privileged to run static code analysis. wtf. Doesn&#x27;t that just mean they run a tool against a binary artefact from a previous job? I fail to see how that requires privileges.
评论 #29973110 未加载
dlor超过 3 年前
This is a great resource. I&#x27;d love to see more reports like it published. CI&#x2F;CD pipelines often run with highly elevated permissions (access to source code, artifact repositories, and production environments), but they are traditionally neglected.
评论 #29966750 未加载
评论 #29972083 未加载
Lucasoato超过 3 年前
Is it just my impression or security in Jenkins seems much more challenging and more time-consuming than in GitLab? This post gives many examples where GitLab was attacked, so of course bad practices like privileged containers can lead to the compromise of a server independently by the technology used, but from my experience with Jenkins, I&#x27;ve seen using passwords in plaintext so many times, even in big companies.
评论 #29968716 未加载
评论 #29965917 未加载
评论 #29965941 未加载
评论 #29965939 未加载
评论 #29967576 未加载
评论 #29965955 未加载
rawgabbit超过 3 年前
You would think by now we would have better credential methods. I still see username and passwords for system credentials. I see tokens created by three legged auths. I don’t get how that is an improvement. The problem is that most deployed code doesn’t have just one credential but a dozen. Multiply that with several environments and you get security fatigue and apathy.
mvdwoord超过 3 年前
The company I currently do contract work for, decided it would be best to have one large team in Azure DevOps and subdivide all teams in repositories etc with prefixes and homegrown &quot;Governer&quot; scripts, which are enforced in all pipelines.<p>Global find on some terms like &quot;key&quot;, &quot;password&quot; etc were great fun. It really showed most people, our team included, struggled with getting the pipeline to work at all. Let alone doing it in a secure manner.<p>This is a 50k+ employee financial institute. I am honestly surprised these kind of attacks are not much more widespread.
tialaramex超过 3 年前
A recurring theme is that they obtain <i>secret</i> credentials from a service which needs to verify credentials, and then turn around and use those to impersonate the entity providing those credentials. For example getting Jenkins to run some Groovy discovers credentials Jenkins uses to verify who is accessing it, and then you can just use those credentials yourself.<p>To fix this - almost anywhere - stop using shared secrets. Every time you visit a (HTTPS) web site, you are provided with the credentials to verify its identity. But, you don&#x27;t gain the ability to impersonate the site because they&#x27;re not <i>secret</i> credentials, they&#x27;re <i>public</i>. You can and should use this in a few places in typical CI &#x2F; CD type infrastructure today, and we should be encouraging other services to enable it too ASAP.<p>In a few places they mention MFA. Again, most MFA involves secrets, for example TOTP Relying Parties need to know what code you should be typing in, so, they need the seed from which to generate that code, and attackers can steal that seed. WebAuthn doesn&#x27;t involve secrets, so, attackers who steal WebAuthn credentials don&#x27;t achieve anything. Unfortunately chances are you enabled one or more vulnerable credential types &quot;just in case&quot;...
jiggawatts超过 3 年前
A weakness of modern secret management is that it isn’t.<p>A secret value ought to be very carefully guarded <i>even from the host machine itself</i>.<p>.NET for example has SecureString, which is a good start — it can’t be accidentally printed or serialised insecurely. If it is serialised, then it is automatically encrypted by the host OS data protection API.<p>Windows even has TPM-hosted certificates! They’re essentially a smart card plugged into the motherboard.<p>A running app can <i>use</i> a TPM credential to sign requests but it can’t read or copy it.<p>These advancements are just completely ignored in the UNIX world, where everything is blindly copied into easily accessible locations in plain text…
评论 #29972510 未加载
mdoms超过 3 年前
&gt; The credentials gave the NCC Group consultant access as a limited user to the Jenkins Master web login UI which was only accessible internally and not from the Internet. After a couple of clicks and looking around in the cluster they were able to switch to an administrator account.<p>These kinds of statements are giving major &quot;draw the rest of the owl&quot; vibes.<p><a href="https:&#x2F;&#x2F;i.kym-cdn.com&#x2F;photos&#x2F;images&#x2F;newsfeed&#x2F;000&#x2F;572&#x2F;078&#x2F;d6d.jpg" rel="nofollow">https:&#x2F;&#x2F;i.kym-cdn.com&#x2F;photos&#x2F;images&#x2F;newsfeed&#x2F;000&#x2F;572&#x2F;078&#x2F;d6d...</a>
movedx超过 3 年前
This is exactly why I both love and hate CI&#x2F;CD.<p>Ultimately most CI&#x2F;CD setups are basically systems administrators with privileged access to everything, network connected and running 24&#x2F;7. It&#x27;s pretty dangerous stuff.<p>I don&#x27;t have an answer though, expect maybe to keep the CI and CD in separate, isolated instances that require manual intervention to bridge the gap on a case by case basis. That doesn&#x27;t scale very well though.
评论 #29972193 未加载
评论 #29972531 未加载
thomasmarcelis超过 3 年前
&gt;&gt;In our final scenario, the NCC Group consultant got booked on a scenario-based assessment:<p>&gt;&gt;“Pretend you have compromised a developer’s laptop.”<p>Most companies will fail right here. Especially outside of the tech world security hygiene with developer&#x27;s laptops is very bad from what I have seen.
cerved超过 3 年前
&gt; a hardcoded git command with a credential was revealed<p><i>cries in security</i>
评论 #29965669 未加载