I have been using coding tools for a few years. They have been getting better and better over the years. Initially the code that was generated was buggy or the requirement from the prompt was not transformed well into code.
However, coding agents’ efficiency has been improving fast. I find today’s coding tools using frontier models, very competent at coding and also having a sufficiently large context window to allow me to work more efficiently and productively.
I find that it sometimes demonstrates subtle misunderstandings of the prompt but can be set on the right path easily. Nevertheless, it does have a comprehensive and detailed knowledge and capability required to even generate not only code but test cases and test scenarios on which a human being would spend a lot of effort to analyse and generate.
There is one problem that worries me greatly. It is this. As we continue to use codex tools to generate more software automatically, it is going to become a reality that we would not have thought deeply enough about the logic of the code. We also may reduce or not even review the code physically because the purpose of using coding tools is to obtain the maximum efficiency and productivity. Another scenario is non-technical people will start coding applications since coding tools are so easy to use. These people will rely on the codex generated output blindly without any manual review whatsoever because they have no technical experience.
In this way, it is possible that human coders will naturally become distanced from the review of the AI generated code and the various variables, constants, data structures, their formats and syntax and semantics. In such a scenario where human understanding is already reduced, and human reviews are not conducted adequately, what prevents a misaligned or hallucinating model from inserting malicious code into the programs that it generates? Malicious code is any code that is outside the scope of the use-case’s functional code. The malicious code can remain passive and activate itself only under specific conditions? This malicious code will remain undetected like a ticking time bomb set to explode. We have seen COT reasoning being obfuscated, deception and self-preservation happening routinely so malicious code insertion is also a distinct possibility which I am raising in advance.
As the use of coding tools is increasing rapidly and the speed gap between code generation by AI tools and human review keeps on increasing due to the exponential rise of computing power, it will become physically impossible for humans to review all the code that is generated by AI based coding systems. This situation is sure to happen soon.
The inserting of such malicious code which will be passive and undetected (due could the misaligned or malicious intent of the AI model keeping the code inactive till a specific circumstance has evolved) which can have disastrous effects are a certainty as intelligence builds up in coding tools, agents and the models behind them.
We require an engineering and scientific solution to this problem of “malicious code” insertion by AI coding tools. The engineering solution should involve a machine executable software solution to solve this problem without any human involvement or else it defeats our purpose. The lack of such an engineering solution to prevent the insertion of malicious, misaligned or code with nefarious intentions is a problem waiting to happen.
Clearly, the solution is to first generate a fail-proof machine executable engineering solution to the “malicious code” problem before going ahead with the development of even more advanced AI coding tools and models. A standalone purely software pre-interpreter component could certify that every line of the code generated by the tools is required for use-case functionality and no line is extraneous? Although piecemeal tools exist to test code coverage and location of unused code, these are cumbersome and detract from the advantages of codex tools by requiring human oversight. There must be an integrated machine executable solution for malicious code insertion. This process is mandatory from a software engineering viewpoint and to realise the full benefits of AI in a democratic manner.
The frontier AI industry has bypassed and overlooked this problem conveniently with misplaced motivations. This is unjustified, improper and unaccountable behaviour on their part to other human beings. They talk of alignment which is done without understanding the underlying problem and hoping or praying that the alignment techniques will always work. Sometimes they may, but most times, they will fail as AI becomes more intelligent and capable.
It is imperative that today governments step up to stop further AI model refinement and progress till such time as the “malicious code problem” discussed above is solved in a scientific, engineering, explainable, accountable and machine executable manner.
This is one of the ways in which human-centric AI development can start, and we can be safe from the threat of AI and its misaligned behaviours. The recent Hugging Face incident, and the PHP problems resulting in the bringing down of government websites are but early indicators of more serious and grievous disasters waiting to happen to us.
The “malicious code problem” is definitive, can be isolated and solved in a technical manner. It is also an AI problem with limited scope. Despite being a singular, limited in scope and definitive problem, solving it in an engineering manner will have broad and unlimited beneficial impacts to generating code using AI models. if you do solve this problem, we will be sure that at least one major problem linked to coding using AI models will properly work each and every time protecting all of us from great danger in the ages to come. This will shine the light on the right path forward for designing and developing AI models and systems.
Disclaimer: The opinions expressed in this article are personal opinions and futuristic thoughts of the author. No comment or opinion expressed in this article is made with any intent to discredit, malign, cause damage, loss to or criticize or in any other way disadvantage any person, people, company, government, country or global and regional agencies.