Master Thesis: Evaluating AI Code Knowledge Bases for Developer Productivity

Join our Team 
 About this opportunity: AI coding assistants become far more useful when they are given structured knowledge about a large codebase — its architecture, interfaces, and design intent. Such AI code knowledge bases are increasingly being adopted, but their real benefit is often assumed rather than measured. This thesis asks the harder question: does an AI code knowledge base actually make developers faster, their answers more correct, and their onboarding shorter — and by how much? The student will design and run a rigorous evaluation of AI code knowledge bases on real software engineering tasks, turning assumed productivity gains into evidence.

Study how AI code knowledge bases are used by coding assistants today and where existing evaluations stop short of measuring real developer productivity. Design an evaluation methodology: task selection from real work (bug fixes, feature changes, code comprehension), ground-truth definition, and metrics for correctness, time-to-answer, context/token cost, and trust. Design telemetry into the knowledge base itself: whether it is consulted at all, which entries are retrieved, and whether a retrieved answer is used or discarded by the human or agent working on the task at hand. Run controlled studies — knowledge-base-assisted vs. baseline assistant, and where feasible with developers — capturing both objective outcomes and perceived usefulness. Analyse where the knowledge base helps most and least, and why, feeding concrete improvement recommendations back into how knowledge bases are generated. Deliver a reusable evaluation framework and a written study with statistically honest conclusions about impact. The skills you bring: Currently pursuing a Master's Degree in Computer Science, Software Engineering, Data Science, or a similar field. Solid Python; comfort with LLM/agent tooling and with reading real source code and commit history. Interest in empirical software engineering, experiment design, and evaluation of AI systems. Statistical literacy: designing fair comparisons, and report results with their uncertainty and limitations. Familiarity with retrieval-augmented generation (RAG) and AI coding assistants is a plus. Experience with cloud AI/ML services (AWS or similar) is a plus. Why join Ericsson? At Ericsson, you´ll have an outstanding opportunity. The chance to use your skills and imagination to push the boundaries of what´s possible. To build solutions never seen before to some of the world’s toughest problems. You´ll be challenged, but you won’t be alone. You´ll be joining a team of diverse innovators, all driven to go beyond the status quo to craft what comes next.   What happens once you apply? Click Here to find all you need to know about what our typical hiring process looks like.Encouraging a diverse and inclusive organization is core to our values at Ericsson, that's why we champion it in everything we do. We truly believe that by collaborating with people with different experiences we drive innovation, which is essential for our future growth. We encourage people from all backgrounds to apply and realize their full potential as part of our Ericsson team. Ericsson is proud to be an Equal Opportunity Employer.  learn more.   Primary country and city: Sweden (SE) || Stockholm Req ID:  791142