Audit Methods Explained
This topic engages with the different audit methodologies and suppliers
Traditional Company Audit:
This method involves approaching a professional auditing firm, receiving a formal quote, and having a dedicated team assigned to conduct the audit. It's characterized by a high level of formality and structured process. I intentionally also include “decentralized” firms with a pool of independent auditors.
Has one large problem which is the scaling factor, the firm wants to grow which means they need to hire more auditors, often sub-optimal auditors are then hired which cannot provide the same value as the senior auditors that have built the great name.
The past has shown that even the most reputable companies have performed poor audits.
Ideal for: Large projects with sufficient budgets that prioritize thoroughness, professionalism, and potential compliance with future web3 regulations.
Solo Auditor:
Solo audits are conducted by individual auditors. They are less formal and usually more flexible in terms of scheduling and requirements.
Ideal for: Smaller projects or startups with limited budgets, where the primary focus is cost-efficiency and quick turnaround times. Additionally can be conducted as initial audits or final clean up audits.
Audit Contest (Code4rena, Sherlock, ..):
Audit contests involve multiple auditors or teams competing to find vulnerabilities in a smart contract. This method tends to attract a diverse range of talent and perspectives.
Ideal for: Projects with medium to large budgets that require a broad sweep of potential issues and can afford to conduct post-contest mitigation.
Static Analyzer Company:
This is a company which is solely focused on marketing, audit reports often yield automated findings which are common upon static analyzer tools. No real security value is provided. This report is an example: https://pvp.money/PVP_Audit.pdf
Ideal for: Nothing, pure trash, avoid at all costs even if you get paid for letting them audit your codebase.
Provider Analysis: Pros and Cons
Traditional Audit
Pros:
- Access to a team of auditors, often with ideal skillsets.
- Professional management of the audit process.
- Benefits from corporate marketing and potential future regulatory compliance.
Cons:
- High cost.
- Uncertainty about the specific auditors assigned. This is the point where most firms fail, simply due to scaling purposes which then requires them to assign less-optimal auditors. Remember, really great auditors are still rare.
Key Consideration: Ensure clarity on the auditors’ qualifications and experience.
Solo Auditor
Pros:
- Cost-effective.
- More direct communication and less formal process.
- Skill level can be verified, though it requires discernment to not be deceived by great marketing skills.
Cons:
- Variable auditor skill levels, supply of great auditors is low.
- Limited resources and lack of synergy compared to other methods.
- May not comply with future regulatory standards, most of the time its a private service and not a regulated entity.
Key Consideration: Validate the auditor's expertise beyond their marketing presence.
Contest / Competitive Audit
Pros:
- High participation leading to detection of many common issues.
- Quick scheduling.
- Effective for a broad vulnerability sweep.
Cons:
- May miss deeper, complex issues, contests are often with sub-optimal deadlines and auditors might focus just on certain parts which then results in non-identified issues with regards to cross-contract / larger scope issues.
- Can be expensive.
- Senior auditors might be scarce, experienced auditors almost often switch to the private route
- Lack of post-audit resolution, needs a full new audit.
Key Consideration: Effective when combined with other methods.
Budget-Friendly Strategy
Incorporate a seasoned researcher into the development team for an internal audit, followed by an external review from another experienced individual. Compensating with tokens can be a cost-effective strategy.
Ideal Audit Strategy
Conclusion
As you can see, there are many approaches and it is clear that no approach is perfect, everything here has its pro’s and con’s, but that's life and you have to deal with it.
How we would to handle it
1. Approach an audit company, ensure that you their best auditors, preferably these are already known via contests, public media or past experiences. This will yield the first formal audit which ideally mitigates all issues and also serves as potential certificate for the future if regulations arrive. And yes, they will come, there will be a day where it’s illegal to deploy a contract without having a certified audit.
2. Approach a solo-auditor to conduct a follow-up audit with the target to specifically focus on deep logical bugs.
3. Thoroughness and talent costs time, this will reflect in the quote - if you get a cheap quote the possibility is high that insufficient resources will be allocated.
This is what we think the best strategy to get the most quality for your “buck”.
This topic engages with the different audit methodologies and suppliers
Traditional Company Audit:
This method involves approaching a professional auditing firm, receiving a formal quote, and having a dedicated team assigned to conduct the audit. It's characterized by a high level of formality and structured process. I intentionally also include “decentralized” firms with a pool of independent auditors.
Has one large problem which is the scaling factor, the firm wants to grow which means they need to hire more auditors, often sub-optimal auditors are then hired which cannot provide the same value as the senior auditors that have built the great name.
The past has shown that even the most reputable companies have performed poor audits.
Ideal for: Large projects with sufficient budgets that prioritize thoroughness, professionalism, and potential compliance with future web3 regulations.
Solo Auditor:
Solo audits are conducted by individual auditors. They are less formal and usually more flexible in terms of scheduling and requirements.
Ideal for: Smaller projects or startups with limited budgets, where the primary focus is cost-efficiency and quick turnaround times. Additionally can be conducted as initial audits or final clean up audits.
Audit Contest (Code4rena, Sherlock, ..):
Audit contests involve multiple auditors or teams competing to find vulnerabilities in a smart contract. This method tends to attract a diverse range of talent and perspectives.
Ideal for: Projects with medium to large budgets that require a broad sweep of potential issues and can afford to conduct post-contest mitigation.
Static Analyzer Company:
This is a company which is solely focused on marketing, audit reports often yield automated findings which are common upon static analyzer tools. No real security value is provided. This report is an example: https://pvp.money/PVP_Audit.pdf
Ideal for: Nothing, pure trash, avoid at all costs even if you get paid for letting them audit your codebase.
Provider Analysis: Pros and Cons
Traditional Audit
Pros:
- Access to a team of auditors, often with ideal skillsets.
- Professional management of the audit process.
- Benefits from corporate marketing and potential future regulatory compliance.
Cons:
- High cost.
- Uncertainty about the specific auditors assigned. This is the point where most firms fail, simply due to scaling purposes which then requires them to assign less-optimal auditors. Remember, really great auditors are still rare.
Key Consideration: Ensure clarity on the auditors’ qualifications and experience.
Solo Auditor
Pros:
- Cost-effective.
- More direct communication and less formal process.
- Skill level can be verified, though it requires discernment to not be deceived by great marketing skills.
Cons:
- Variable auditor skill levels, supply of great auditors is low.
- Limited resources and lack of synergy compared to other methods.
- May not comply with future regulatory standards, most of the time its a private service and not a regulated entity.
Key Consideration: Validate the auditor's expertise beyond their marketing presence.
Contest / Competitive Audit
Pros:
- High participation leading to detection of many common issues.
- Quick scheduling.
- Effective for a broad vulnerability sweep.
Cons:
- May miss deeper, complex issues, contests are often with sub-optimal deadlines and auditors might focus just on certain parts which then results in non-identified issues with regards to cross-contract / larger scope issues.
- Can be expensive.
- Senior auditors might be scarce, experienced auditors almost often switch to the private route
- Lack of post-audit resolution, needs a full new audit.
Key Consideration: Effective when combined with other methods.
Budget-Friendly Strategy
Incorporate a seasoned researcher into the development team for an internal audit, followed by an external review from another experienced individual. Compensating with tokens can be a cost-effective strategy.
Ideal Audit Strategy
- What would I do if money is not a topic?
- Include a solo researcher in the development process for ongoing review.
- Conduct a main audit with a reputable auditing firm.
- Follow up with a contest-style audit to catch any residual issues, ideally finding none.
Conclusion
As you can see, there are many approaches and it is clear that no approach is perfect, everything here has its pro’s and con’s, but that's life and you have to deal with it.
How we would to handle it
1. Approach an audit company, ensure that you their best auditors, preferably these are already known via contests, public media or past experiences. This will yield the first formal audit which ideally mitigates all issues and also serves as potential certificate for the future if regulations arrive. And yes, they will come, there will be a day where it’s illegal to deploy a contract without having a certified audit.
2. Approach a solo-auditor to conduct a follow-up audit with the target to specifically focus on deep logical bugs.
3. Thoroughness and talent costs time, this will reflect in the quote - if you get a cheap quote the possibility is high that insufficient resources will be allocated.
This is what we think the best strategy to get the most quality for your “buck”.
