🤖 Bedrock & S3 Vectors: Smarter Related Posts with AI

One issue we face a lot is that readers often want to discover more content. The AWS Fundamentals Blog has over 100 blog posts. But the issue is, you need to know what you are looking for. A common way of solving this problem is to show posts that are related to the current post somewhere in the article. This is exactly what we are building today.

The End Result Shows Related Posts for Each New Blog Post

Let's start with the end result. If you look on any blog post right now, you should see a section about Related Posts, looking like this:

This shows that the post Localstack Debugging Edge Functions shows the related posts:

1. AWS Edge Locations
2. Render
3. AWS FinOps

Now, how did we build this?

The Architecture Consists of Bedrock & S3 Vectors

The overall architecture consists of just two services:

• Bedrock - for building a knowledge base
• S3 - for Storage & Vectors

Bedrock for Building a Knowledge Base

Bedrock is the service you will use mostly for building AI products on AWS. It gives you access to all the foundational models and developer tooling around it.

AWS's quote in the introduction days of Bedrock was:

The easiest way to build and scale generative AI applications

… and I think it is a great service, but can be quite complex to get started with.

S3 for Storing Content and Vectors

S3 is our classical S3 storage. But we don't use it only for storing data. We also store our vector index in the newly announced service S3 Vectors. This is one of the most exciting "AI" launches AWS had in the recent past.

Building Related Posts from Uploading Content to Creating a Knowledge Base

Let's go ahead and build related posts step-by-step. I'll go into the details of each step on the way.

Step 1 - Preparing and Uploading Content to S3

The first thing we need is our actual content. We use our 100+ articles here as a starting point.

All our blog posts are in markdown format. And each blog post has a frontmatter with some metadata about the post. For example, the post Amazon DSQL Pricing Guide has this frontmatter:

---
title: Amazon Aurora DSQL Pricing Guide
seoTitle: Amazon Aurora DSQL Pricing Guide
seoDescription: Complete guide to Amazon Aurora DSQL pricing including DPU costs, storage fees, free tier limits, and backup pricing. Learn what serverless SQL database costs really look like.
datePublished: 'Fri May 30 2025 21:00:00 GMT+0000 (Coordinated Universal Time)'
slug: amazon-dsql-pricing-guide
cover: /assets/blog/amazon-dsql-pricing-guide/dsql-cover.webp
ogImage: /assets/blog/localstack-debugging-edge-functions/og.webp
tags: >-
    aws-dsql
service: rds
author:
    name: Alessandro Volpicella
    username: sandro_vol
    profilePicture: /assets/authors/sandro_vol.webp
---

# Introduction

The day has come. There's a new serverless database in the house.

After the frontmatter you can see how the content is starting. We will use the data within the frontmatter later on.

For now, we have built a script which is uploading all of that content into a normal S3 bucket. The metadata of each S3 object looks like the following:

You can see that we have the following data in the metadata of each object available:

• Cover - The cover of the post
• Date Published - The date of the post
• SEO Description - A small description of the post
• Service - Which service it belongs to (e.g. Lambda, DynamoDB, etc.)
• Slug - The URL of the post (e.g. amazon-dsql-pricing-guide)
• Tags - Some tags we have given the post like dsql, aws, database, serverless
• Title - The title of the post.

While the metadata is not super crucial here, it is quite useful because you can use it for filtering down your results more clearly. But more on that later.

Step 2 - Creating your Knowledge Base in Bedrock

Now, we need to create a knowledge base in Bedrock. Since we want to use S3 as a vector store, we need to do that manually in the console. Because Terraform and Pulumi are not yet ready for it. See these two issues: Issue 1, Issue 2.

For creating a knowledge base go into Bedrock and open up Knowledge Bases.

In there, click on "Create" and "Knowledge Base with vector store"

Now, you can see 4 steps you need to do.

First, give your knowledge base a name like kb-blog-posts. Next, let it create an IAM role or create one yourself.

The more interesting part comes now with the data source. You have the ability to use various data sources like:

• S3
• Web Crawler
• Custom
• Confluence
• Salesforce
• SharePoint

This already shows the flexibility and integrations Bedrock gives you. We just uploaded all our content to S3, so we use S3 as a data source.

Log Delivery

Important: Please add a log delivery to CloudWatch. I think this should be activated by default (like it is in Lambda functions as well).

This will attach a Log Group to your Knowledge Base, so that you get information about all your syncing processes and failures. If you don't do this you will have no error messages at all in case something is failing.

Step 3 - Data Source & Chunking

Now, we come to the interesting settings. How to set up the chunks of the content.

For that, we need to understand a bit better how a vector store works.

Vector Stores

(feel free to skip this part if you just want to build!)

In a vector store, we want to have a numerical representation of our content. Imagine a simple X and Y axis.

Each post (spoiler: it is actually a chunk, not a post) has its own numerical representation.

The dimensions of our posts could look like that:

• DSQL Pricing: [1.1, 7]
• DynamoDB Streams: [3.5, 6]
• Aurora Introduction: [2, 5]
• Lambda Cold Starts: [7.5, 3.5]
• ECS Cluster Provider ASG: [8.1, 1.5]

The meaning of the X and Y axes is not interesting for us. We only care that each blog post has its own numerical representation.

And the interesting part is now to understand which of those posts are close to each other, and which are far away from each other. This is what we can simply calculate.

In vector stores, you typically use one of the two Distance Metrics:

• Euclidean Distance
• Cosine Distance

Let's calculate two similarities with Cosine Distance:

1. DSQL Pricing - DynamoDB Streams: 0.07
2. DSQL Pricing - Lambda Cold Starts: 0.44

If you want to calculate it yourself the formula is Cosine Distance = 1 - (Dot Product ÷ (Magnitude A × Magnitude B))

That shows that the first pair is very close to each other. The second pair is farther away from each other.

Chunking & Embedding

Now, I mentioned that we don't actually calculate the numerical representation of a post but that we have a chunk of a post. This is because a post is too big to fit into one number (technically). We first need to create several chunks chunk out of the post. And then embed the chunks into numerical representations.

For example, instead of taking the whole pricing guide on DSQL. We take 2,000 character chunks, embed them with an AI model, and put them into the vector store.

This means we don't actually compare the distance from several posts to each other, but the distance of chunks. This comes with some more trade-offs, but it is necessary since you cannot embed a large dataset at once.

There are workarounds, but for our simple example we stay with chunks.

Back to Building

There are several options you can choose in Bedrock now.

Select your Content Bucket first.

Next, the parsing strategy. You can choose to use Bedrock's default parser or use other models as a parser. Since Bedrock's parser is free, we use this one.

Then the chunking strategy. There are several ones like:

• Default Chunking
• Fixed-size chunking
• Hierarchical chunking
• Semantic chunking
• No chunking

We use fixed-size chunking to keep it simple. But for our example it would also make sense to use hierarchical chunking, because our markdown posts have a hierarchy (think of the table of contents). Again, to keep it simple we stick to fixed-size chunking for now.

We use max tokens 2000, and overlap percentage of 10%.

Step 4 - Select Model & Vector Store

Next, we need to select a model. You can use any model you've activated in AWS. We go with the "Titan Text Embeddings V2":

Then, we need to select a vector store.

We select S3 as a vector store. Be aware that this service is still in preview! If you have production workloads, I would not recommend using it.

Then click next & "Create Knowledge Base".

Step 5 - Sync Your Content to Your Knowledge Base

Once your knowledge base is created, you need to initiate a sync. Go to your data source and click on "Start Sync".

Once you start it, you can see how your data is slowly synchronized to the vector store. The whole magic is happening within the sync step.

You can also watch your sync progress using CloudWatch Live Tail:

Step 6 - Test your Knowledge Base with Bedrock

Okay, our knowledge base is now created. Let's test it.

Open up your Bedrock Knowledge Base and click on "Test Knowledge Base":

A new page opens. Select the option "Retrieval only: data sources". This won't feed the responses into an LLM again. It will only give you your data sources based on your query.

Now, I am asking, for example: How expensive is CloudFront.

It then returns with 5 chunks from the article CloudFront Pricing.

If we change the option to "Retrieval and response generation: data sources and model", it will even go one step further and feed it into the LLM you select. The same request then results in a much nicer output:

And this is the actual beauty of it. You now get a reply with a source attached to it.

Metadata

You probably ask yourself now, "Why did we add all of the metadata to each post?" First of all, in the source chunks you can see this metadata as well. This helps you figure out which post, service, etc.

But you can also use the metadata to filter your vector sources. For example, based on your query, which service you want to access. You could filter all of your posts already to give only chunks of the correct services to the LLM. There are many more additions and improvements. We keep those for other posts 😊

Step 7 - Generate Related Posts and Store Them in JSON

Now, just the last step is missing: Actually generating related posts.

As a recap: We want to know which posts are related to each other. For that, we want to show them at the bottom of each post.

There are many "architectural" decisions on how to do that. We went with a pretty simple one.

My idea was to create one JSON file in my repository called generated-posts.json. This file holds all blog posts I have in my repository and an attached array with all the related posts. It looks like this:

{
    "a-comprehensive-guide-to-aws-compute-services": [
        {
            "slug": "social-stats-dashboard-sst-nextjs",
            "title": "How We Built a Social Stats Dashboard Using SST and Next.js",
            "similarity": 0.5818566213733921,
            "service": "sst",
            "datePublished": "Sat Dec 07 2024 11:11:43 GMT+0000 (Coordinated Universal Time)",
            "cover": "/assets/blog/social-stats-dashboard-sst-nextjs/cover.webp"
        },
        {
            "slug": "starter-guide-things-i-wish-i-knew-before-for-aws-lambda",
            "title": "Introduction to AWS Lambda: A Starter Guide",
            "similarity": 0.5719967276903436,
            "service": "lambda",
            "datePublished": "Thu May 27 2021 20:48:12 GMT+0000 (Coordinated Universal Time)",
            "cover": "/assets/blog/starter-guide-things-i-wish-i-knew-before-for-aws-lambda/cover.webp"
        },
        {
            "slug": "aws-lambda-pricing-a-complete-guide-to-understanding-the-cost-of-the-serverless-service",
            "title": "AWS Lambda Pricing: A Complete Guide to Understanding the Cost of the Serverless Service",
            "similarity": 0.5590921875395728,
            "service": "lambda",
            "datePublished": "Wed Jan 25 2023 06:18:16 GMT+0000 (Coordinated Universal Time)",
            "cover": "/assets/blog/aws-lambda-pricing-a-complete-guide-to-understanding-the-cost-of-the-serverless-service/f18396bd-9034-42c6-92e5-b9b4753b704d.webp"
        }
    ]
}

Keep it Simple and Cheap

There are many other opportunities for doing this which are probably a bit more flexible. But since we are just a two-dev team and our blog posts are saved locally in a repository, we thought this is the simplest and most cost-effective solution.

We use Next.js on the frontend. At build time, we regenerate this array if files have changed. And only then do we need to call the Bedrock API and pay.

This also keeps the blog post page very fast without any loading indicators. It is just HTML which lies in a CloudFront CDN cache somewhere.

Script to generate related posts

The script for creating that JSON was a bit back and forth. Not all posts have related posts, but we still want to show them. So what we did is the following:

Improve Your Knowledge Base with Chunking, Preparation, and Better Queries

This is it! We have quite some room for improvement. We also save the results while generating related posts to see how many matches we have. And this is what we saw:

{
    "totalQueries": 244,
    "mainQueries": 136,
    "fallbackQueries": 0,
    "averageResultsPerMainQuery": 1.4705882352941178,
    "averageResultsPerFallbackQuery": 0,
    "postsWithNoResults": 30
}

That means we only have 1.4 related posts per actual query. And even 30 posts without any related post.

What we could improve:

• Better chunking: Use a transformer Lambda and use hierarchical chunking
• Better data preparation: Get a good summary of the post for each post
• Build a better query: The query we generate the posts from is pretty basic; improve that
• Use filtering on metadata: Make it a 2-step process. First, figure out the service, then ask the knowledge base

With AI products, but also with every product, we are developing in an iterative way. That means don't try to have it perfect from the beginning. Execute it end to end and improve from there.