This should work:

// untested
public class HostNameToUpperLayout : LayoutSkeleton
{
    public override void ActivateOptions()
    {
        // empty
    }

    public override void Format(TextWriter writer, LoggingEvent loggingEvent)
    {
        string hostName = (string)loggingEvent.LookupProperty("log4net:HostName");
        writer.Write(hostName.ToUpper());
    }
}

<parameter>
      <parameterName value="@message"/>
      <dbType value="String"/>
      <size value="4000"/>
      <layout type="Company.Application.HostNameToUpperLayout, Company.Application" />
</parameter>

There are more clever ways of doing it but that should work.

Whoops, I re-read your post and saw that you wanted to output both the hostname and the message at the same time. In that case you'd probably want to continue using the PatternLayout and register a special converter:

public class HostNameToUpperConverter : PatternConverter
{
    protected override void Convert(TextWriter writer, object state)
    {
        string hostName = (string)GlobalContext.Properties[LoggingEvent.HostNameProperty]
        writer.Write(hostName.ToUpper());
    }
}

<layout type="log4net.Layout.PatternLayout">
        <converter>
            <name value="hostNameToUpper" />
            <type value="ConsoleApplication1.HostNameToUpperConverter, ConsoleApplication1" />
        </converter>
        <conversionPattern value="[%hostNameToUpper] - %message" />
</layout>