Arrange 25 crackers in a single layer on the bottom of a parchment paper lined 9x13-inch pan.
Add butter, brown sugar, and granulated sugar into a saucepan over medium heat and bring to a simmer, stirring often. Add in the graham cracker crumbs and milk, then stir to combine.
Remove from heat and pour half of the sugar mixture over the crackers and top with 25 more crackers. Pour the remaining sugar mixture over the second layer of crackers, then top with remaining crackers.
Add chocolate chips and peanut butter into a microwave safe bowl. Microwave in 30-second intervals, stirring in between, until melted and smooth.
Pour the melted chocolate over the top layer of crackers and spread it evenly all the way to the edges.
Refrigerate until set, at least 1 hour. Once it is completely cooled, slice into small bars.