Posting here just to keep this knowledge somewhere. To make dh-make-perl work with Module::Build based Perl module distros you need:
$(PERL) Makefile.PL INSTALLDIRS=vendor $(MAKE) OPTIMIZE="$(OPTIMIZE)" LD_RUN_PATH=""with
$(PERL) Build.PL installdirs=vendor OPTIMIZE="$(OPTIMIZE)" LD_RUN_PATH="" ./Buildand
$(MAKE) install PREFIX=$(TMP)/usrwith
./Build install destdir=$(TMP)